Sep 06 10:30:14.857362 ip-10-0-130-119 systemd[1]: Starting Kubernetes Kubelet...
Sep 06 10:30:15.288622 ip-10-0-130-119 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:30:15.288622 ip-10-0-130-119 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 06 10:30:15.288622 ip-10-0-130-119 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:30:15.288622 ip-10-0-130-119 kubenswrapper[2573]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 06 10:30:15.288622 ip-10-0-130-119 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 06 10:30:15.290627 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.290498    2573 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295441    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295462    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295466    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295469    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295473    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295475    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.295468 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295479    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295482    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295485    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295487    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295490    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295493    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295495    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295498    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295500    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295503    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295506    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295509    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295511    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295514    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295516    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295519    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295523    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295527    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295531    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.295725 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295533    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295536    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295538    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295541    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295543    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295546    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295555    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295558    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295561    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295563    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295565    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295568    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295570    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295572    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295576    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295579    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295582    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295584    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295587    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295589    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.296245 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295592    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295596    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295600    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295603    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295606    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295609    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295612    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295618    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295621    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295624    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295626    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295629    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295631    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295634    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295636    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295638    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295641    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295644    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295647    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.296749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295649    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295652    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295654    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295657    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295659    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295662    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295665    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295667    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295670    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295673    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295675    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295678    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295680    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295683    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295685    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295687    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295691    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295693    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295696    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295699    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.297239 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295701    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295703    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295706    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295708    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295711    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295713    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295716    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295719    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295721    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295724    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295727    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295729    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295732    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295734    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.295737    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297450    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297458    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297462    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297465    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297468    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.297722 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297470    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297473    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297476    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297478    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297480    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297484    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297489    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297492    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297494    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297496    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297499    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297502    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297506    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297509    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297512    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297515    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297517    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297520    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297523    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.298219 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297526    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297529    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297531    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297534    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297537    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297540    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297542    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297545    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297547    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297550    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297552    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297555    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297557    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297560    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297562    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297565    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297567    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297570    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297572    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297575    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.298693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297577    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297580    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297582    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297586    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297589    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297591    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297594    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297597    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297601    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297604    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297607    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297609    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297611    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297615    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297619    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297622    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297624    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297627    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297631    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.299208 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297635    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297638    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297641    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297644    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297646    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297649    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297652    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297654    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297657    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297659    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297662    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297664    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297667    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297669    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297672    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297675    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297677    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297680    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297682    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297685    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.299674 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297688    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297691    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297693    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297695    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297700    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297702    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297705    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297707    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297710    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297712    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297715    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297717    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297720    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297723    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297725    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.297728    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298151    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298157    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298159    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298162    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.300207 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298165    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298168    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298170    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298173    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298176    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298179    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298181    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298184    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298187    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298189    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298192    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298195    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298204    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298207    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298210    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298212    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298215    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298217    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298220    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298223    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.300693 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298226    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298228    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298232    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298235    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298238    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298241    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298243    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298246    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298248    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298251    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298253    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298257    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298259    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298263    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298266    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298268    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298272    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298274    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298277    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.301194 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298279    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298282    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298284    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298287    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298289    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298292    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298300    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298303    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298305    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298307    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298310    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298312    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298315    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298317    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298320    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298322    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298324    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298327    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298329    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298331    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.301670 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298334    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298337    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298339    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298343    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298346    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298348    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298351    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298353    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298355    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298358    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298361    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298363    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298366    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298368    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298371    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298375    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298378    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298381    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298384    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.302164 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298394    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298397    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298399    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298402    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298404    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298406    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298409    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298411    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298414    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298416    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298418    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298421    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298423    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298426    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298430    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298433    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.298436    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298512    2573 flags.go:64] FLAG: --address="0.0.0.0"
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298547    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298556    2573 flags.go:64] FLAG: --anonymous-auth="true"
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298560    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 06 10:30:15.302631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298564    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298568    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298572    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298577    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298580    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298583    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298586    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298589    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298592    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298595    2573 flags.go:64] FLAG: --cgroup-root=""
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298598    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298601    2573 flags.go:64] FLAG: --client-ca-file=""
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298610    2573 flags.go:64] FLAG: --cloud-provider="external"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298613    2573 flags.go:64] FLAG: --cluster-dns="[]"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298620    2573 flags.go:64] FLAG: --cluster-domain=""
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298623    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298626    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298629    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298632    2573 flags.go:64] FLAG: --container-log-max-files="5"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298636    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298639    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298642    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298645    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298648    2573 flags.go:64] FLAG: --contention-profiling="false"
Sep 06 10:30:15.303194 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298652    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298654    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298657    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298660    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298664    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298667    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298670    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298673    2573 flags.go:64] FLAG: --enable-load-reader="false"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298676    2573 flags.go:64] FLAG: --enable-server="true"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298678    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298685    2573 flags.go:64] FLAG: --event-burst="100"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298688    2573 flags.go:64] FLAG: --event-qps="50"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298691    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298693    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298696    2573 flags.go:64] FLAG: --eviction-hard=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298700    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298703    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298706    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298709    2573 flags.go:64] FLAG: --eviction-soft=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298712    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298714    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298723    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298725    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298728    2573 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298731    2573 flags.go:64] FLAG: --fail-swap-on="true"
Sep 06 10:30:15.303771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298733    2573 flags.go:64] FLAG: --feature-gates=""
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298737    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298740    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298743    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298746    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298749    2573 flags.go:64] FLAG: --healthz-port="10248"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298752    2573 flags.go:64] FLAG: --help="false"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298755    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298758    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298760    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298763    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298766    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298770    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298773    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298775    2573 flags.go:64] FLAG: --image-service-endpoint=""
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298778    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298784    2573 flags.go:64] FLAG: --kube-api-burst="100"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298786    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298789    2573 flags.go:64] FLAG: --kube-api-qps="50"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298792    2573 flags.go:64] FLAG: --kube-reserved=""
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298795    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298798    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298801    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298803    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 06 10:30:15.304395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298806    2573 flags.go:64] FLAG: --lock-file=""
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298808    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298811    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298814    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298819    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298823    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298825    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298828    2573 flags.go:64] FLAG: --logging-format="text"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298831    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298834    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298837    2573 flags.go:64] FLAG: --manifest-url=""
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298840    2573 flags.go:64] FLAG: --manifest-url-header=""
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298844    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298847    2573 flags.go:64] FLAG: --max-open-files="1000000"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298851    2573 flags.go:64] FLAG: --max-pods="110"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298855    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298858    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298861    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298864    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298866    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298869    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298872    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298879    2573 flags.go:64] FLAG: --node-status-max-images="50"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298883    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298888    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 06 10:30:15.304992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298891    2573 flags.go:64] FLAG: --pod-cidr=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298893    2573 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298898    2573 flags.go:64] FLAG: --pod-manifest-path=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298901    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298904    2573 flags.go:64] FLAG: --pods-per-core="0"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298907    2573 flags.go:64] FLAG: --port="10250"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298910    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298913    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e7bc713c092feb6f"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298916    2573 flags.go:64] FLAG: --qos-reserved=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298919    2573 flags.go:64] FLAG: --read-only-port="10255"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298922    2573 flags.go:64] FLAG: --register-node="true"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298925    2573 flags.go:64] FLAG: --register-with-taints=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298928    2573 flags.go:64] FLAG: --registry-burst="10"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298932    2573 flags.go:64] FLAG: --registry-qps="5"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298935    2573 flags.go:64] FLAG: --reserved-cpus=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298937    2573 flags.go:64] FLAG: --reserved-memory=""
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298941    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298944    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298947    2573 flags.go:64] FLAG: --rotate-certificates="false"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298949    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298952    2573 flags.go:64] FLAG: --runonce="false"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298955    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298958    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298962    2573 flags.go:64] FLAG: --seccomp-default="false"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298965    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 06 10:30:15.305644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298968    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298971    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298974    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298977    2573 flags.go:64] FLAG: --storage-driver-password="root"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298980    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298983    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298987    2573 flags.go:64] FLAG: --storage-driver-user="root"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298990    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298993    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298996    2573 flags.go:64] FLAG: --system-cgroups=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.298998    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299003    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299006    2573 flags.go:64] FLAG: --tls-cert-file=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299009    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299013    2573 flags.go:64] FLAG: --tls-min-version=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299016    2573 flags.go:64] FLAG: --tls-private-key-file=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299018    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299039    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299044    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299048    2573 flags.go:64] FLAG: --v="2"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299052    2573 flags.go:64] FLAG: --version="false"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299059    2573 flags.go:64] FLAG: --vmodule=""
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299063    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299067    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299199    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.306347 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299203    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299207    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299210    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299213    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299217    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299223    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299225    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299228    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299230    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299233    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299235    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299239    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299243    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299247    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299250    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299252    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299255    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299258    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299260    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.306952 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299263    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299265    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299268    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299270    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299273    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299275    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299277    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299280    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299282    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299285    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299288    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299290    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299292    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299295    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299297    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299300    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299302    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299305    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299308    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299311    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.307430 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299314    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299316    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299318    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299321    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299324    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299326    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299330    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299332    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299335    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299337    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299340    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299342    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299345    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299348    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299350    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299352    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299355    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299357    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299360    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299362    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.307911 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299365    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299367    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299369    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299372    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299375    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299377    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299380    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299382    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299385    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299387    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299391    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299393    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299395    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299398    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299400    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299403    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299405    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299408    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299412    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299414    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.308418 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299417    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299419    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299422    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299424    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299427    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299429    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299432    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299434    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299436    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299440    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299443    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299446    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299449    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299451    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299455    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299458    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299461    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299464    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.308921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.299467    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.299471    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.307747    2573 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.307775    2573 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.307798    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.307803    2573 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307865    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307870    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307874    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307877    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307880    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307883    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307886    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307888    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307891    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.309499 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307893    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307896    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307898    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307901    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307904    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307906    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307909    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307911    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307914    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307916    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307920    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307922    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307925    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307927    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307930    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307933    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307936    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307938    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307941    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307944    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.309933 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307953    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307956    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307959    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307961    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307964    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307967    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307969    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307972    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307976    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307980    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307983    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307986    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307989    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307992    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307995    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.307998    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308001    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308004    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308007    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.310442 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308010    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308014    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308016    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308019    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308037    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308041    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308045    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308048    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308051    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308054    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308057    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308059    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308062    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308065    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308074    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308077    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308079    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308081    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308085    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308088    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.310889 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308091    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308094    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308096    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308099    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308101    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308104    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308106    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308109    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308112    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308115    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308117    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308120    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308123    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308125    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308128    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308130    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308133    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308135    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308138    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308140    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.311405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308143    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308145    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308148    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308151    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308153    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308156    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308158    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308166    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308169    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308171    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308174    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.308179    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308324    2573 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308329    2573 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308332    2573 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 06 10:30:15.311946 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308335    2573 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308338    2573 feature_gate.go:328] unrecognized feature gate: Example
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308340    2573 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308343    2573 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308346    2573 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308348    2573 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308353    2573 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308356    2573 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308359    2573 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308362    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308365    2573 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308368    2573 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308370    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308373    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308375    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308378    2573 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308380    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308383    2573 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308386    2573 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308388    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 06 10:30:15.312336 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308391    2573 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308393    2573 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308395    2573 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308398    2573 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308401    2573 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308408    2573 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308411    2573 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308413    2573 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308416    2573 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308418    2573 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308421    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308423    2573 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308426    2573 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308429    2573 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308431    2573 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308434    2573 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308437    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308439    2573 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308442    2573 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308444    2573 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 06 10:30:15.312919 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308447    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308449    2573 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308452    2573 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308454    2573 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308456    2573 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308459    2573 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308462    2573 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308464    2573 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308466    2573 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308469    2573 feature_gate.go:328] unrecognized feature gate: Example2
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308472    2573 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308474    2573 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308477    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308479    2573 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308482    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308484    2573 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308487    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308489    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308497    2573 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 06 10:30:15.313581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308500    2573 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308503    2573 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308505    2573 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308507    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308510    2573 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308512    2573 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308515    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308517    2573 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308520    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308522    2573 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308525    2573 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308527    2573 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308530    2573 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308532    2573 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308534    2573 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308537    2573 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308539    2573 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308542    2573 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308544    2573 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308547    2573 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 06 10:30:15.314069 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308549    2573 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308552    2573 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308555    2573 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308559    2573 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308562    2573 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308564    2573 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308567    2573 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308569    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308572    2573 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308575    2573 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308577    2573 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308579    2573 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308588    2573 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308590    2573 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308593    2573 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308596    2573 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 06 10:30:15.314609 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:15.308600    2573 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.308605    2573 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.308750    2573 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.311373    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.312479    2573 server.go:1019] "Starting client certificate rotation"
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.312597    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:30:15.315296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.313435    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 06 10:30:15.315567 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.315386    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:30:15.320372 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.320344    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 06 10:30:15.335078 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.335054    2573 log.go:25] "Validated CRI v1 runtime API"
Sep 06 10:30:15.338525 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.338505    2573 log.go:25] "Validated CRI v1 image API"
Sep 06 10:30:15.339436 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.339416    2573 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 06 10:30:15.342942 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.342923    2573 fs.go:135] Filesystem UUIDs: map[0adea259-9d2f-475e-adca-cb636b13467c:/dev/nvme0n1p4 0e30e82d-0e8f-44af-b4ed-6d1fd1fc7566:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 06 10:30:15.343001 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.342942    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 06 10:30:15.344824 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.344654    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:30:15.347297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.347178    2573 manager.go:216] Machine: {Timestamp:2026-09-06 10:30:15.346671403 +0000 UTC m=+0.376657376 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092709 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24a4b430df45628846d254e5ed64c8 SystemUUID:ec24a4b4-30df-4562-8846-d254e5ed64c8 BootID:ced6aeb9-8c9a-46e9-b95b-2650bc84e7b6 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:42:6f:27:0c:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:42:6f:27:0c:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:4b:90:e7:0e:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 06 10:30:15.347297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.347292    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 06 10:30:15.347409 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.347377    2573 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 06 10:30:15.350937 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.350907    2573 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 06 10:30:15.351108 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.350941    2573 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-119.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 06 10:30:15.352578 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.352568    2573 topology_manager.go:138] "Creating topology manager with none policy"
Sep 06 10:30:15.352616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.352582    2573 container_manager_linux.go:309] "Creating device plugin manager"
Sep 06 10:30:15.352616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.352590    2573 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:30:15.353466 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.353455    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 06 10:30:15.355497 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.355485    2573 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 06 10:30:15.360183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360170    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:30:15.360326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360316    2573 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 06 10:30:15.360364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360348    2573 kubelet.go:486] "Attempting to sync node with API server"
Sep 06 10:30:15.360364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360361    2573 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 06 10:30:15.360417 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360379    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 06 10:30:15.360417 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360392    2573 kubelet.go:398] "Adding apiserver pod source"
Sep 06 10:30:15.360417 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360408    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 06 10:30:15.360862 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360844    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:30:15.360966 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.360869    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 06 10:30:15.363418 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.363396    2573 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 06 10:30:15.366414 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366395    2573 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 06 10:30:15.366460 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366431    2573 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 06 10:30:15.366606 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366595    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 06 10:30:15.366638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366615    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 06 10:30:15.366638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366624    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 06 10:30:15.366638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366630    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 06 10:30:15.366638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366638    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 06 10:30:15.366739 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.366644    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 06 10:30:15.367247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367234    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 06 10:30:15.367247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367245    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367253    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367258    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367277    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367286    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367265    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6kl2l"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367310    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 06 10:30:15.367373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.367316    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 06 10:30:15.371337 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.371323    2573 server.go:1268] "Started kubelet"
Sep 06 10:30:15.371477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.371394    2573 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 06 10:30:15.372107 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.372018    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 06 10:30:15.372192 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.372168    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 06 10:30:15.372252 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.372200    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 06 10:30:15.372252 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.372164    2573 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-119.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 06 10:30:15.372252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.372171    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-119.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 06 10:30:15.372341 ip-10-0-130-119 systemd[1]: Started Kubernetes Kubelet.
Sep 06 10:30:15.375833 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.375811    2573 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 06 10:30:15.377100 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.377074    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6kl2l"
Sep 06 10:30:15.377278 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.377251    2573 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 06 10:30:15.377765 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.377748    2573 server.go:310] "Adding debug handlers to kubelet server"
Sep 06 10:30:15.380687 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.379532    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-119.ec2.internal.18d2b4a34647cfca  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-119.ec2.internal,UID:ip-10-0-130-119.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-119.ec2.internal,},FirstTimestamp:2026-09-06 10:30:15.371288522 +0000 UTC m=+0.401274496,LastTimestamp:2026-09-06 10:30:15.371288522 +0000 UTC m=+0.401274496,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-119.ec2.internal,}"
Sep 06 10:30:15.381630 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.381614    2573 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 06 10:30:15.381700 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.381639    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 06 10:30:15.381825 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.381807    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.382447 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.382428    2573 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 06 10:30:15.382685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.382467    2573 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 06 10:30:15.382685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.382425    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 06 10:30:15.382685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.382644    2573 reconstruct.go:97] "Volume reconstruction finished"
Sep 06 10:30:15.382685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.382653    2573 reconciler.go:29] "Reconciler: start to sync state"
Sep 06 10:30:15.383070 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383053    2573 factory.go:55] Registering systemd factory
Sep 06 10:30:15.383070 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383073    2573 factory.go:223] Registration of the systemd container factory successfully
Sep 06 10:30:15.383279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383269    2573 factory.go:153] Registering CRI-O factory
Sep 06 10:30:15.383279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383279    2573 factory.go:223] Registration of the crio container factory successfully
Sep 06 10:30:15.383354 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383322    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 06 10:30:15.383354 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383342    2573 factory.go:103] Registering Raw factory
Sep 06 10:30:15.383354 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383355    2573 manager.go:1196] Started watching for new ooms in manager
Sep 06 10:30:15.383720 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.383708    2573 manager.go:349] Starting recovery of all containers
Sep 06 10:30:15.392539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.392519    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:15.394619 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.394603    2573 manager.go:354] Recovery completed
Sep 06 10:30:15.395261 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.395239    2573 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.398873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.398862    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.405681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.405660    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.405782 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.405697    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.405782 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.405712    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.406331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.406313    2573 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 06 10:30:15.406331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.406329    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.406463 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.406329    2573 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 06 10:30:15.406463 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.406431    2573 state_mem.go:36] "Initialized new in-memory state store"
Sep 06 10:30:15.407090 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.407077    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.407137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.407102    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.407137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.407114    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.408576 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.408563    2573 policy_none.go:49] "None policy: Start"
Sep 06 10:30:15.408619 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.408581    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 06 10:30:15.408619 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.408591    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 06 10:30:15.410606 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.410596    2573 policy_none.go:47] "Start"
Sep 06 10:30:15.452754 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.452719    2573 manager.go:337] "Starting Device Plugin manager"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.452787    2573 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.452801    2573 server.go:87] "Starting device plugin registration server"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.453163    2573 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.453176    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.453284    2573 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.453376    2573 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.453385    2573 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.453919    2573 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 06 10:30:15.466318 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.453977    2573 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.484593 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.484556    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 06 10:30:15.485872 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.485851    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 06 10:30:15.485989 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.485905    2573 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 06 10:30:15.485989 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.485929    2573 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 06 10:30:15.485989 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.485964    2573 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 06 10:30:15.487859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.487835    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:15.554278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.554159    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.555340 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.555324    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.555401 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.555354    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.555401 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.555364    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.555401 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.555389    2573 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.563362 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.563344    2573 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.563457 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.563368    2573 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-119.ec2.internal\": node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.576332 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.576308    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.586631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.586606    2573 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"]
Sep 06 10:30:15.586716 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.586690    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.588500 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.588475    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.588602 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.588511    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.588602 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.588526    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.590911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.590898    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.591055 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.591019    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.591112 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.591076    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.592745 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592731    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.592854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592757    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.592854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592767    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.592854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592735    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.592854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592835    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.592854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.592847    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.595182 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.595165    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.595240 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.595202    2573 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 06 10:30:15.596018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.596001    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientMemory"
Sep 06 10:30:15.596133 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.596042    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasNoDiskPressure"
Sep 06 10:30:15.596133 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.596054    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeHasSufficientPID"
Sep 06 10:30:15.614652 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.614631    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.619091 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.619073    2573 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-119.ec2.internal\" not found" node="ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.676401 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.676366    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.776678 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.776640    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.783921 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.783890    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.784051 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.783928    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.784051 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.783954    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d150e533eb692c08b346a2806d019c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"d150e533eb692c08b346a2806d019c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.877377 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.877310    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:15.884628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884606    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.884681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.884681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d150e533eb692c08b346a2806d019c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"d150e533eb692c08b346a2806d019c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.884746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884708    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.884746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82a76ebfaacc16dd96304698dc2cac0e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal\" (UID: \"82a76ebfaacc16dd96304698dc2cac0e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.884804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.884749    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d150e533eb692c08b346a2806d019c7d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-119.ec2.internal\" (UID: \"d150e533eb692c08b346a2806d019c7d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.917814 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.917773    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.921520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:15.921502    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:15.978348 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:15.978308    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.078906 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:16.078869    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.179426 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:16.179354    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.280112 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:16.280071    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.312621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.312594    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 06 10:30:16.313124 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.312776    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:30:16.313124 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.312785    2573 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 06 10:30:16.380799 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:16.380771    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.383522 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.383476    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-05 10:25:17 +0000 UTC" deadline="2028-03-19 08:13:16.389359155 +0000 UTC"
Sep 06 10:30:16.383612 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.383523    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13437h43m0.005840826s"
Sep 06 10:30:16.480920 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:16.480896    2573 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-119.ec2.internal\" not found"
Sep 06 10:30:16.487542 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:16.487493    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd150e533eb692c08b346a2806d019c7d.slice/crio-953c4a9ca027d7923130aba6a742cc1a1b544179d0dc3c9da1064de45964b961 WatchSource:0}: Error finding container 953c4a9ca027d7923130aba6a742cc1a1b544179d0dc3c9da1064de45964b961: Status 404 returned error can't find the container with id 953c4a9ca027d7923130aba6a742cc1a1b544179d0dc3c9da1064de45964b961
Sep 06 10:30:16.488303 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:16.488280    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a76ebfaacc16dd96304698dc2cac0e.slice/crio-025e047d66edc4688f08be3f0e7ce11dbcc0b530a11f911bf05c99938396610b WatchSource:0}: Error finding container 025e047d66edc4688f08be3f0e7ce11dbcc0b530a11f911bf05c99938396610b: Status 404 returned error can't find the container with id 025e047d66edc4688f08be3f0e7ce11dbcc0b530a11f911bf05c99938396610b
Sep 06 10:30:16.492403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.492386    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:30:16.529129 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.529094    2573 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:16.582140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.582112    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:16.593570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.593547    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:30:16.595066 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.595050    2573 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal"
Sep 06 10:30:16.602659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:16.602629    2573 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 06 10:30:18.813018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:18.812989    2573 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:19.221120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.221079    2573 apiserver.go:52] "Watching apiserver"
Sep 06 10:30:19.227977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.227951    2573 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 06 10:30:19.235300 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.235267    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rzvjc","openshift-multus/multus-6zhpb","openshift-multus/multus-additional-cni-plugins-kzp48","openshift-multus/network-metrics-daemon-dkvt2","kube-system/konnectivity-agent-wsb9h","openshift-dns/node-resolver-hc7gh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal","openshift-network-diagnostics/network-check-target-hf4tq","openshift-network-operator/iptables-alerter-bwsbz","openshift-ovn-kubernetes/ovnkube-node-nr7sl","kube-system/global-pull-secret-syncer-n764h","kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq","openshift-cluster-node-tuning-operator/tuned-2nrm6"]
Sep 06 10:30:19.237776 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.237755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.240628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.240608    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.241106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241087    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 06 10:30:19.241234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241108    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.241306 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241244    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 06 10:30:19.241364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241313    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5gptd\""
Sep 06 10:30:19.241537 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241518    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 06 10:30:19.241688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241667    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 06 10:30:19.241756 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.241715    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.242565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.242541    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.242657 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.242566    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 06 10:30:19.242657 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.242645    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bwn8j\""
Sep 06 10:30:19.243649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.242823    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 06 10:30:19.243649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.242827    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.243649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.243142    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.245112 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.245090    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 06 10:30:19.245238 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.245140    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8r82j\""
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.246526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.246593    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.246998    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.247113    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.247117    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.247196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.247178    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 06 10:30:19.249326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.249308    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.251896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.251413    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kf56d\""
Sep 06 10:30:19.251896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.251604    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 06 10:30:19.251896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.251784    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 06 10:30:19.252491 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.252640 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252624    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-systemd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252713 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252667    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252713 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252684    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-netd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252713 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-device-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.252836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-system-cni-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.252836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252731    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-node-log\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-log-socket\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252770    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovn-node-metrics-cert\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.252836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252816    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252853    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-bin\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252915    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9bp\" (UniqueName: \"kubernetes.io/projected/d614386c-c370-4e35-8055-1bddfbaf8242-kube-api-access-dg9bp\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.252974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-cnibin\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253001    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-systemd-units\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253023    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-netns\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-var-lib-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-ovn\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253097    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-config\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253156    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27cgc\" (UniqueName: \"kubernetes.io/projected/b249010b-f2cd-4131-b7d3-0146ae3c3922-kube-api-access-27cgc\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253280    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5plnq\" (UniqueName: \"kubernetes.io/projected/631ae72d-88b6-4777-8e72-53906c337bb2-kube-api-access-5plnq\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253337    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-etc-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253385    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-registration-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253442    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-sys-fs\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-kubelet\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-env-overrides\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253541    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-script-lib\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.253760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253563    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-socket-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.253760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.253586    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-slash\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.254616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.254594    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.254749 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.254699    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-g722g\""
Sep 06 10:30:19.254837 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.254819    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.255134 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.255119    2573 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 06 10:30:19.256706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.256454    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.258353 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.258332    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.258655 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.258638    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rc7zw\""
Sep 06 10:30:19.259257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.258836    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 06 10:30:19.259257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.259008    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.259257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.259122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.263197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.262299    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.263197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.263028    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.263348 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.263303    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.265308 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.265285    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zhgbb\""
Sep 06 10:30:19.265792 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.265773    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 06 10:30:19.266018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.266002    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bmjwh\""
Sep 06 10:30:19.268717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.268695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.271235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.270892    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 06 10:30:19.271235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.271008    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 06 10:30:19.271235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.270932    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-v2mjb\""
Sep 06 10:30:19.271235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.271201    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 06 10:30:19.272979 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.272959    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.273080 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.273055    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:19.275377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.275362    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:19.275440 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.275425    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:19.279962 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.279938    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jsqs2"
Sep 06 10:30:19.286220 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.286194    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jsqs2"
Sep 06 10:30:19.343690 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.343661    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 06 10:30:19.351011 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.350957    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"82a76ebfaacc16dd96304698dc2cac0e","Type":"ContainerStarted","Data":"025e047d66edc4688f08be3f0e7ce11dbcc0b530a11f911bf05c99938396610b"}
Sep 06 10:30:19.352107 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.352080    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" event={"ID":"d150e533eb692c08b346a2806d019c7d","Type":"ContainerStarted","Data":"953c4a9ca027d7923130aba6a742cc1a1b544179d0dc3c9da1064de45964b961"}
Sep 06 10:30:19.354313 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354290    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.354408 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354323    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-bin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.354408 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354356    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-etc-kubernetes\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.354408 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354393    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-node-log\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354437    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-node-log\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354490    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-log-socket\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354540    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-log-socket\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354613    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354640    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-modprobe-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-kubernetes\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.354706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354691    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354715    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-var-lib-kubelet\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354721    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81643241-0445-4393-9b5c-028981d2dfe0-serviceca\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-multus\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdfd\" (UniqueName: \"kubernetes.io/projected/b95d18fd-4d41-449d-89e8-dc637b17d5bd-kube-api-access-fkdfd\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354840    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354894    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-multus-certs\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354948    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-systemd-units\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.354971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.354972    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-var-lib-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355001    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-ovn\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355027    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355048    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-systemd-units\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355057    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-var-lib-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355094    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/281a9014-bf0e-409e-8d9f-42982ba2595e-agent-certs\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355105    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355069    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-ovn\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355096    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-multus-daemon-config\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-run\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-cni-binary-copy\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvmls\" (UniqueName: \"kubernetes.io/projected/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-kube-api-access-vvmls\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.355392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355415    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-slash\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-netd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355468    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-slash\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355484    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-device-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-netd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355513    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-lib-modules\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355542    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-host\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355563    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-device-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-socket-dir-parent\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355613    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-hostroot\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355643    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-conf-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-system-cni-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovn-node-metrics-cert\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355729    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-system-cni-dir\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355754    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/281a9014-bf0e-409e-8d9f-42982ba2595e-konnectivity-ca\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355783    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysconfig\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355806    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzk6\" (UniqueName: \"kubernetes.io/projected/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-kube-api-access-2gzk6\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355839    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-cnibin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355871    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-kubelet\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-bin\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9bp\" (UniqueName: \"kubernetes.io/projected/d614386c-c370-4e35-8055-1bddfbaf8242-kube-api-access-dg9bp\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.355989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-sys\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356007    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s2z2\" (UniqueName: \"kubernetes.io/projected/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-kube-api-access-8s2z2\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-cnibin\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-tuned\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-tmp\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95d18fd-4d41-449d-89e8-dc637b17d5bd-hosts-file\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356235    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-kubelet-config\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356252    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-dbus\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-netns\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356307    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-config\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.356795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27cgc\" (UniqueName: \"kubernetes.io/projected/b249010b-f2cd-4131-b7d3-0146ae3c3922-kube-api-access-27cgc\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-sys-fs\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-systemd\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356384    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356402    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5plnq\" (UniqueName: \"kubernetes.io/projected/631ae72d-88b6-4777-8e72-53906c337bb2-kube-api-access-5plnq\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-cni-bin\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-etc-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356498    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-registration-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356504    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-etc-openvswitch\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356547    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-iptables-alerter-script\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356579    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-conf\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356576    2573 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356606    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-etc-selinux\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356610    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81643241-0445-4393-9b5c-028981d2dfe0-host\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356646    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-kubelet\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.357579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356656    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-netns\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-env-overrides\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-script-lib\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356692    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-socket-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356718    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7srt\" (UniqueName: \"kubernetes.io/projected/81643241-0445-4393-9b5c-028981d2dfe0-kube-api-access-g7srt\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356743    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d18fd-4d41-449d-89e8-dc637b17d5bd-tmp-dir\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-system-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356786    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-k8s-cni-cncf-io\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-systemd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-host-slash\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356861    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-netns\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.356887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncpxr\" (UniqueName: \"kubernetes.io/projected/971fbef9-6745-4123-88a4-fc2c9649b359-kube-api-access-ncpxr\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357157    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-config\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357288    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-env-overrides\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357423    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovnkube-script-lib\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/631ae72d-88b6-4777-8e72-53906c337bb2-cnibin\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.358296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-registration-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357706    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-socket-dir\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-run-ovn-kubernetes\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357791    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-run-systemd\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b249010b-f2cd-4131-b7d3-0146ae3c3922-host-kubelet\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.357847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d614386c-c370-4e35-8055-1bddfbaf8242-sys-fs\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.358859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.358003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/631ae72d-88b6-4777-8e72-53906c337bb2-cni-binary-copy\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.360839 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.360820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b249010b-f2cd-4131-b7d3-0146ae3c3922-ovn-node-metrics-cert\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.367881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.367857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27cgc\" (UniqueName: \"kubernetes.io/projected/b249010b-f2cd-4131-b7d3-0146ae3c3922-kube-api-access-27cgc\") pod \"ovnkube-node-nr7sl\" (UID: \"b249010b-f2cd-4131-b7d3-0146ae3c3922\") " pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.368114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.368097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9bp\" (UniqueName: \"kubernetes.io/projected/d614386c-c370-4e35-8055-1bddfbaf8242-kube-api-access-dg9bp\") pod \"aws-ebs-csi-driver-node-d2nlq\" (UID: \"d614386c-c370-4e35-8055-1bddfbaf8242\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.368326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.368301    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5plnq\" (UniqueName: \"kubernetes.io/projected/631ae72d-88b6-4777-8e72-53906c337bb2-kube-api-access-5plnq\") pod \"multus-additional-cni-plugins-kzp48\" (UID: \"631ae72d-88b6-4777-8e72-53906c337bb2\") " pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.457460 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457424    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-cni-binary-copy\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.457460 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457471    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvmls\" (UniqueName: \"kubernetes.io/projected/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-kube-api-access-vvmls\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.457706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.457706 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.457604    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:19.457816 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.457710    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:19.957671162 +0000 UTC m=+3.127521025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:19.457816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-lib-modules\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457817    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-host\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457842    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-socket-dir-parent\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457860    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-hostroot\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-conf-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/281a9014-bf0e-409e-8d9f-42982ba2595e-konnectivity-ca\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457901    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-host\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.457928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457907    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysconfig\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-hostroot\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457957    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzk6\" (UniqueName: \"kubernetes.io/projected/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-kube-api-access-2gzk6\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457969    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysconfig\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457977    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-socket-dir-parent\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457984    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-conf-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-cnibin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.457981    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-lib-modules\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458028    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-kubelet\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-sys\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458076    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-kubelet\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s2z2\" (UniqueName: \"kubernetes.io/projected/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-kube-api-access-8s2z2\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-sys\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-tuned\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458178    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-tmp\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458182    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-cnibin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458202    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95d18fd-4d41-449d-89e8-dc637b17d5bd-hosts-file\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458226    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-kubelet-config\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.458294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458265    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-cni-binary-copy\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458297    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-dbus\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458324    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-systemd\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458341    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b95d18fd-4d41-449d-89e8-dc637b17d5bd-hosts-file\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458351    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-iptables-alerter-script\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458375    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-conf\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458379    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-kubelet-config\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458400    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7508edae-7d98-4088-a13f-ac16080cdf40-dbus\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458458    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-systemd\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81643241-0445-4393-9b5c-028981d2dfe0-host\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458520    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81643241-0445-4393-9b5c-028981d2dfe0-host\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458548    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/281a9014-bf0e-409e-8d9f-42982ba2595e-konnectivity-ca\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458549    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7srt\" (UniqueName: \"kubernetes.io/projected/81643241-0445-4393-9b5c-028981d2dfe0-kube-api-access-g7srt\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.458986 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.458575    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-conf\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.459516 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459192    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d18fd-4d41-449d-89e8-dc637b17d5bd-tmp-dir\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.459649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459497    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-iptables-alerter-script\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.459738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459701    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-system-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.459799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459743    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-k8s-cni-cncf-io\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.459799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-host-slash\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.459893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459814    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-netns\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.459893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncpxr\" (UniqueName: \"kubernetes.io/projected/971fbef9-6745-4123-88a4-fc2c9649b359-kube-api-access-ncpxr\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.459893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459875    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.460037 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459875    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-system-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460037 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459949    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-k8s-cni-cncf-io\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460037 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.459999    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-host-slash\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.460203 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460050    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-netns\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460203 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b95d18fd-4d41-449d-89e8-dc637b17d5bd-tmp-dir\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.460310 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-bin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460310 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-etc-kubernetes\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460411 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460411 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-modprobe-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460411 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460358    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-bin\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460541 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460523    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-multus-cni-dir\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460599 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-kubernetes\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460668 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460668 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-tuned\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460761 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460662    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-var-lib-kubelet\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460761 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.460692    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:19.460761 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460724    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-etc-kubernetes\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.460761 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.460757    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:19.960738731 +0000 UTC m=+3.130588593 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460782    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-sysctl-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460789    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-modprobe-d\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-var-lib-kubelet\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460848    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-etc-kubernetes\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460850    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81643241-0445-4393-9b5c-028981d2dfe0-serviceca\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.460900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-multus\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdfd\" (UniqueName: \"kubernetes.io/projected/b95d18fd-4d41-449d-89e8-dc637b17d5bd-kube-api-access-fkdfd\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460973    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-var-lib-cni-multus\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.460979    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-tmp\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-multus-certs\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461049    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461083    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/281a9014-bf0e-409e-8d9f-42982ba2595e-agent-certs\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461112    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-multus-daemon-config\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.461201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-run\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.461574 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461325    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-run\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.461574 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461368    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/81643241-0445-4393-9b5c-028981d2dfe0-serviceca\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.461574 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461383    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/971fbef9-6745-4123-88a4-fc2c9649b359-host-run-multus-certs\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.461747 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.461726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/971fbef9-6745-4123-88a4-fc2c9649b359-multus-daemon-config\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.464259 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.464234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/281a9014-bf0e-409e-8d9f-42982ba2595e-agent-certs\") pod \"konnectivity-agent-wsb9h\" (UID: \"281a9014-bf0e-409e-8d9f-42982ba2595e\") " pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.466607 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.466561    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzk6\" (UniqueName: \"kubernetes.io/projected/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-kube-api-access-2gzk6\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.467196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.467179    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7srt\" (UniqueName: \"kubernetes.io/projected/81643241-0445-4393-9b5c-028981d2dfe0-kube-api-access-g7srt\") pod \"node-ca-rzvjc\" (UID: \"81643241-0445-4393-9b5c-028981d2dfe0\") " pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.467530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.467505    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s2z2\" (UniqueName: \"kubernetes.io/projected/f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be-kube-api-access-8s2z2\") pod \"tuned-2nrm6\" (UID: \"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be\") " pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.468076 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.468056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvmls\" (UniqueName: \"kubernetes.io/projected/d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5-kube-api-access-vvmls\") pod \"iptables-alerter-bwsbz\" (UID: \"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5\") " pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.470755 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.470735    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncpxr\" (UniqueName: \"kubernetes.io/projected/971fbef9-6745-4123-88a4-fc2c9649b359-kube-api-access-ncpxr\") pod \"multus-6zhpb\" (UID: \"971fbef9-6745-4123-88a4-fc2c9649b359\") " pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.473203 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.473145    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:19.473203 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.473182    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:19.473203 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.473192    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:19.473362 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.473263    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:19.973244291 +0000 UTC m=+3.143094161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:19.473536 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.473521    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdfd\" (UniqueName: \"kubernetes.io/projected/b95d18fd-4d41-449d-89e8-dc637b17d5bd-kube-api-access-fkdfd\") pod \"node-resolver-hc7gh\" (UID: \"b95d18fd-4d41-449d-89e8-dc637b17d5bd\") " pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.550362 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.550319    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:19.557843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.557810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq"
Sep 06 10:30:19.559724 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.559697    2573 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:19.572585 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.572561    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kzp48"
Sep 06 10:30:19.581279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.581256    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:19.590791 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.590771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hc7gh"
Sep 06 10:30:19.599244 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.599223    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bwsbz"
Sep 06 10:30:19.606862 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.606840    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2nrm6"
Sep 06 10:30:19.611630 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.611611    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6zhpb"
Sep 06 10:30:19.617300 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.617277    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rzvjc"
Sep 06 10:30:19.759403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.759327    2573 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 06 10:30:19.965430 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.965397    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:19.965588 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:19.965446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:19.965588 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.965562    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:19.965588 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.965564    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:19.965724 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.965621    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:20.965607133 +0000 UTC m=+4.135457001 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:19.965724 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:19.965636    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:20.965629355 +0000 UTC m=+4.135479205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:20.066118 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.066039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:20.066301 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.066233    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:20.066301 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.066263    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:20.066301 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.066275    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:20.066440 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.066349    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:21.06632811 +0000 UTC m=+4.236177981 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:20.289414 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.289380    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:25:19 +0000 UTC" deadline="2028-05-24 09:44:11.052855708 +0000 UTC"
Sep 06 10:30:20.297690 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.289439    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15023h13m50.763423503s"
Sep 06 10:30:20.316196 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.316112    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971fbef9_6745_4123_88a4_fc2c9649b359.slice/crio-a85177f8ccc1b61435c403d21bf5149f81689264c5fa9d065ba9df4e301f7004 WatchSource:0}: Error finding container a85177f8ccc1b61435c403d21bf5149f81689264c5fa9d065ba9df4e301f7004: Status 404 returned error can't find the container with id a85177f8ccc1b61435c403d21bf5149f81689264c5fa9d065ba9df4e301f7004
Sep 06 10:30:20.317206 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.317180    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281a9014_bf0e_409e_8d9f_42982ba2595e.slice/crio-e3e36d1b45c63af723a0f0bccbae017c1d4f886b224a673ea3919f2dc73e7b13 WatchSource:0}: Error finding container e3e36d1b45c63af723a0f0bccbae017c1d4f886b224a673ea3919f2dc73e7b13: Status 404 returned error can't find the container with id e3e36d1b45c63af723a0f0bccbae017c1d4f886b224a673ea3919f2dc73e7b13
Sep 06 10:30:20.318229 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.318209    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd052ce88_8d66_4bd1_beff_f9a0eb6c8ab5.slice/crio-c931ad7b3b8be48d4c6d547aa5ae372e21c13eb84be29f74408e6e1dce850228 WatchSource:0}: Error finding container c931ad7b3b8be48d4c6d547aa5ae372e21c13eb84be29f74408e6e1dce850228: Status 404 returned error can't find the container with id c931ad7b3b8be48d4c6d547aa5ae372e21c13eb84be29f74408e6e1dce850228
Sep 06 10:30:20.321482 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.319762    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd614386c_c370_4e35_8055_1bddfbaf8242.slice/crio-73cc86b67ead24e29e0de8589beeb496d7cd9d5656b4aaa0818176b17da2c6c3 WatchSource:0}: Error finding container 73cc86b67ead24e29e0de8589beeb496d7cd9d5656b4aaa0818176b17da2c6c3: Status 404 returned error can't find the container with id 73cc86b67ead24e29e0de8589beeb496d7cd9d5656b4aaa0818176b17da2c6c3
Sep 06 10:30:20.321482 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.321041    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6dbb774_c2ea_41e4_bd78_2fdc0f15d2be.slice/crio-187cf2e5c411f36fc1d9db281ae85720d8b5e4543be9436c53c052bf723862e8 WatchSource:0}: Error finding container 187cf2e5c411f36fc1d9db281ae85720d8b5e4543be9436c53c052bf723862e8: Status 404 returned error can't find the container with id 187cf2e5c411f36fc1d9db281ae85720d8b5e4543be9436c53c052bf723862e8
Sep 06 10:30:20.350409 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.350385    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb249010b_f2cd_4131_b7d3_0146ae3c3922.slice/crio-fff8224cc0b1d4fbf7dccd750ffe9306b2b57bbd26a91cec5875f04ba08f7791 WatchSource:0}: Error finding container fff8224cc0b1d4fbf7dccd750ffe9306b2b57bbd26a91cec5875f04ba08f7791: Status 404 returned error can't find the container with id fff8224cc0b1d4fbf7dccd750ffe9306b2b57bbd26a91cec5875f04ba08f7791
Sep 06 10:30:20.351632 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.351487    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81643241_0445_4393_9b5c_028981d2dfe0.slice/crio-2315c4ec7385e36a03c94fdc817ebfc450742371dc4fc140c39f12b6413526b2 WatchSource:0}: Error finding container 2315c4ec7385e36a03c94fdc817ebfc450742371dc4fc140c39f12b6413526b2: Status 404 returned error can't find the container with id 2315c4ec7385e36a03c94fdc817ebfc450742371dc4fc140c39f12b6413526b2
Sep 06 10:30:20.351910 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.351886    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb95d18fd_4d41_449d_89e8_dc637b17d5bd.slice/crio-14ce80428a3777d08f5d63122dc3a26983aa9cf2ec625c6f2fb8df5f9d289e94 WatchSource:0}: Error finding container 14ce80428a3777d08f5d63122dc3a26983aa9cf2ec625c6f2fb8df5f9d289e94: Status 404 returned error can't find the container with id 14ce80428a3777d08f5d63122dc3a26983aa9cf2ec625c6f2fb8df5f9d289e94
Sep 06 10:30:20.353271 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:30:20.353248    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631ae72d_88b6_4777_8e72_53906c337bb2.slice/crio-dccb713c1d5dc29e2a459a728dd28356ffa526797c1fbd0d1f53a46ca5212ad3 WatchSource:0}: Error finding container dccb713c1d5dc29e2a459a728dd28356ffa526797c1fbd0d1f53a46ca5212ad3: Status 404 returned error can't find the container with id dccb713c1d5dc29e2a459a728dd28356ffa526797c1fbd0d1f53a46ca5212ad3
Sep 06 10:30:20.354876 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.354590    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hc7gh" event={"ID":"b95d18fd-4d41-449d-89e8-dc637b17d5bd","Type":"ContainerStarted","Data":"14ce80428a3777d08f5d63122dc3a26983aa9cf2ec625c6f2fb8df5f9d289e94"}
Sep 06 10:30:20.356628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.356605    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzvjc" event={"ID":"81643241-0445-4393-9b5c-028981d2dfe0","Type":"ContainerStarted","Data":"2315c4ec7385e36a03c94fdc817ebfc450742371dc4fc140c39f12b6413526b2"}
Sep 06 10:30:20.357784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.357761    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nrm6" event={"ID":"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be","Type":"ContainerStarted","Data":"187cf2e5c411f36fc1d9db281ae85720d8b5e4543be9436c53c052bf723862e8"}
Sep 06 10:30:20.358813 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.358793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"fff8224cc0b1d4fbf7dccd750ffe9306b2b57bbd26a91cec5875f04ba08f7791"}
Sep 06 10:30:20.360011 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.359983    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wsb9h" event={"ID":"281a9014-bf0e-409e-8d9f-42982ba2595e","Type":"ContainerStarted","Data":"e3e36d1b45c63af723a0f0bccbae017c1d4f886b224a673ea3919f2dc73e7b13"}
Sep 06 10:30:20.361685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.361651    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6zhpb" event={"ID":"971fbef9-6745-4123-88a4-fc2c9649b359","Type":"ContainerStarted","Data":"a85177f8ccc1b61435c403d21bf5149f81689264c5fa9d065ba9df4e301f7004"}
Sep 06 10:30:20.363146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.363124    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq" event={"ID":"d614386c-c370-4e35-8055-1bddfbaf8242","Type":"ContainerStarted","Data":"73cc86b67ead24e29e0de8589beeb496d7cd9d5656b4aaa0818176b17da2c6c3"}
Sep 06 10:30:20.365430 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.365413    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwsbz" event={"ID":"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5","Type":"ContainerStarted","Data":"c931ad7b3b8be48d4c6d547aa5ae372e21c13eb84be29f74408e6e1dce850228"}
Sep 06 10:30:20.972720 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.972676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:20.972914 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:20.972762    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:20.972983 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.972922    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:20.973034 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.972987    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:22.972968472 +0000 UTC m=+6.142818326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:20.973456 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.973434    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:20.973562 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:20.973492    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:22.973476527 +0000 UTC m=+6.143326384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:21.073736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.073695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:21.083958 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.083390    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:21.083958 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.083429    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:21.083958 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.083449    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:21.083958 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.083531    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:23.083495673 +0000 UTC m=+6.253345536 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:21.290448 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.290329    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-05 10:25:19 +0000 UTC" deadline="2028-02-25 17:58:27.773857942 +0000 UTC"
Sep 06 10:30:21.290448 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.290396    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12895h28m6.483466285s"
Sep 06 10:30:21.348973 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.348934    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:21.349153 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.349094    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:21.349575 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.349557    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:21.349685 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.349659    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:21.349755 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.349730    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:21.349812 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:21.349796    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:21.375831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.375737    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerStarted","Data":"dccb713c1d5dc29e2a459a728dd28356ffa526797c1fbd0d1f53a46ca5212ad3"}
Sep 06 10:30:21.401738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.401641    2573 generic.go:356] "Generic (PLEG): container finished" podID="82a76ebfaacc16dd96304698dc2cac0e" containerID="5f8a189548faf84fae02eaf0d631efa850d6d330a1383d5f5ec002cc78d9987a" exitCode=0
Sep 06 10:30:21.401738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.401760    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"82a76ebfaacc16dd96304698dc2cac0e","Type":"ContainerDied","Data":"5f8a189548faf84fae02eaf0d631efa850d6d330a1383d5f5ec002cc78d9987a"}
Sep 06 10:30:21.426603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:21.426083    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" event={"ID":"d150e533eb692c08b346a2806d019c7d","Type":"ContainerStarted","Data":"53c9c5f3ada7f6fe3d61e75294fb67f39c8ddec96a99d75b3dcf1a54cf1956d0"}
Sep 06 10:30:22.453273 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:22.453135    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" event={"ID":"82a76ebfaacc16dd96304698dc2cac0e","Type":"ContainerStarted","Data":"9eb30c85ea60d81d5821484c25f5c14cff807487f6e670b088eac07105dca2f1"}
Sep 06 10:30:22.468009 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:22.467908    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-119.ec2.internal" podStartSLOduration=4.467890139 podStartE2EDuration="4.467890139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:21.440124135 +0000 UTC m=+4.609974008" watchObservedRunningTime="2026-09-06 10:30:22.467890139 +0000 UTC m=+5.637740012"
Sep 06 10:30:22.989270 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:22.989229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:22.989452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:22.989318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:22.989555 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:22.989528    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:22.989624 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:22.989598    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:26.989577542 +0000 UTC m=+10.159427415 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:22.990049 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:22.990027    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:22.990184 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:22.990125    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:26.990107545 +0000 UTC m=+10.159957397 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:23.090505 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:23.090460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:23.090767 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.090736    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:23.090767 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.090765    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:23.090918 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.090779    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:23.090918 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.090846    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:27.090825152 +0000 UTC m=+10.260675030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:23.348368 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:23.348283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:23.348556 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.348453    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:23.348874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:23.348853    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:23.348983 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.348961    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:23.349145 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:23.349129    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:23.349277 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:23.349252    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:25.346674    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:25.346840    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:25.347245    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:25.347357    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:25.347417    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:25.347569 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:25.347501    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:27.028151 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.027898    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:27.028631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.028221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:27.028631 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.028048    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:27.028631 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.028330    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:35.028309342 +0000 UTC m=+18.198159208 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:27.028631 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.028343    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:27.028631 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.028376    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:35.028365393 +0000 UTC m=+18.198215242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:27.128973 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.128935    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:27.129180 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.129093    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:27.129180 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.129110    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:27.129180 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.129119    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:27.129348 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.129194    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:35.12917834 +0000 UTC m=+18.299028210 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:27.347539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.347455    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:27.347539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.347476    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:27.347748 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.347573    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:27.347748 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:27.347599    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:27.347748 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.347662    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:27.347748 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:27.347717    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:29.346896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:29.346858    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:29.346896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:29.346893    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:29.347369 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:29.346857    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:29.347369 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:29.347001    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:29.347369 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:29.347047    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:29.347369 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:29.347107    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:31.349922 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:31.349889    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:31.350387 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:31.350021    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:31.350387 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:31.350050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:31.350387 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:31.350040    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:31.350387 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:31.350151    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:31.350387 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:31.350230    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:33.349550    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:33.349584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:33.349553    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:33.349686    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:33.349770    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:33.350030 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:33.349869    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:35.091852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.091758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:35.091852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.091821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:35.092310 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.091936    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:35.092310 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.091936    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:35.092310 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.091999    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:51.091983173 +0000 UTC m=+34.261833045 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:35.092310 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.092014    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:30:51.092007374 +0000 UTC m=+34.261857225 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:35.192364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.192318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:35.192560 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.192535    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:35.192638 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.192567    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:35.192638 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.192580    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:35.192723 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.192637    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:30:51.192615813 +0000 UTC m=+34.362465663 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:35.347118 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.347041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:35.347276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.347041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:35.347276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:35.347186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:35.347276 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.347186    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:35.347397 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.347272    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:35.347397 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:35.347348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:37.351196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:37.351148    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:37.351708 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:37.351278    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:37.351708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:37.351334    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:37.351708 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:37.351428    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:37.351708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:37.351493    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:37.351708 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:37.351558    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:39.347455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.347137    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:39.348106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.347177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:39.348106 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:39.347526    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:39.348106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.347173    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:39.348106 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:39.347586    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:39.348106 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:39.347633    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:39.489538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.489507    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq" event={"ID":"d614386c-c370-4e35-8055-1bddfbaf8242","Type":"ContainerStarted","Data":"9619bfa5229db82c0fe690a78e61aeadd41b6a77f6c020ecfd7e884fd770604e"}
Sep 06 10:30:39.490827 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.490800    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hc7gh" event={"ID":"b95d18fd-4d41-449d-89e8-dc637b17d5bd","Type":"ContainerStarted","Data":"3a36a4b71b8d7575210102a5b8f74378ee7c6538cd8d9bd3e9a03e5d365b7041"}
Sep 06 10:30:39.492290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.492264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rzvjc" event={"ID":"81643241-0445-4393-9b5c-028981d2dfe0","Type":"ContainerStarted","Data":"d414cf37494dcee4225f8fbe51f9cc4c378ec94691f78b5ae968435a8dc2aab6"}
Sep 06 10:30:39.493865 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.493826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2nrm6" event={"ID":"f6dbb774-c2ea-41e4-bd78-2fdc0f15d2be","Type":"ContainerStarted","Data":"bbc93e3020d73491eec9f5d362cd8b59d4cd93b2bbd680d365a65f87ca9f1aa8"}
Sep 06 10:30:39.497021 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.496995    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"ae8780138a385afc5767d420212daf8bd805bba86c429610589cf7a5593c0b3c"}
Sep 06 10:30:39.497128 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.497059    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"f9b3da94ae0b0d34f13566462f3ae63e7d9e82c13e1df846039b33ae757a4f1b"}
Sep 06 10:30:39.497128 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.497098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"460989e2f1bed757398544a3f666f902a226c8fffe946a4ada3e668f498bca8d"}
Sep 06 10:30:39.497264 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.497144    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"6b40d41ddcf95eaa9cd6e5260bd7c68943a591f94bb49417ce3319e0ea764e63"}
Sep 06 10:30:39.497264 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.497192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"b1174a24a27547b336b3727629d384adcfed562cf7fe41b45c26430892fda837"}
Sep 06 10:30:39.497264 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.497228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"094807f1fc17bded06983eec59958f8a60f2f20af8e21e7e70837464e2b8a8f5"}
Sep 06 10:30:39.498452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.498430    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wsb9h" event={"ID":"281a9014-bf0e-409e-8d9f-42982ba2595e","Type":"ContainerStarted","Data":"09eb916c5351373bd3e04f2f9c7ec0951fb8483b51e51aa3b1c768c3b5ed0999"}
Sep 06 10:30:39.499751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.499731    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6zhpb" event={"ID":"971fbef9-6745-4123-88a4-fc2c9649b359","Type":"ContainerStarted","Data":"c6adb67f7ed64c2f42d5b7f789bfd1c279aa8236ccdb557175b279d102fa88ef"}
Sep 06 10:30:39.501221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.501192    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="ce2f937771ea8083c643903f1a56ac02f91552022ea94344f1a87296231b4d5b" exitCode=0
Sep 06 10:30:39.501324 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.501229    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"ce2f937771ea8083c643903f1a56ac02f91552022ea94344f1a87296231b4d5b"}
Sep 06 10:30:39.505565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.505529    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hc7gh" podStartSLOduration=4.391578533 podStartE2EDuration="22.505517732s" totalImagesPullingTime="18.113939199s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:39.505026475 +0000 UTC m=+22.674876347" watchObservedRunningTime="2026-09-06 10:30:39.505517732 +0000 UTC m=+22.675367604"
Sep 06 10:30:39.505638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.505591    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-119.ec2.internal" podStartSLOduration=21.505587632 podStartE2EDuration="21.505587632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:22.468826346 +0000 UTC m=+5.638676218" watchObservedRunningTime="2026-09-06 10:30:39.505587632 +0000 UTC m=+22.675437504"
Sep 06 10:30:39.537438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.537390    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wsb9h" podStartSLOduration=4.416916754 podStartE2EDuration="22.537370016s" totalImagesPullingTime="18.120453262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:39.536845959 +0000 UTC m=+22.706695831" watchObservedRunningTime="2026-09-06 10:30:39.537370016 +0000 UTC m=+22.707219889"
Sep 06 10:30:39.551729 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.551688    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2nrm6" podStartSLOduration=4.426358873 podStartE2EDuration="22.551675686s" totalImagesPullingTime="18.125316813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:39.551354329 +0000 UTC m=+22.721204200" watchObservedRunningTime="2026-09-06 10:30:39.551675686 +0000 UTC m=+22.721525557"
Sep 06 10:30:39.566620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.566577    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6zhpb" podStartSLOduration=4.360675764 podStartE2EDuration="22.566563831s" totalImagesPullingTime="18.205888067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:39.566182234 +0000 UTC m=+22.736032097" watchObservedRunningTime="2026-09-06 10:30:39.566563831 +0000 UTC m=+22.736413703"
Sep 06 10:30:39.578384 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:39.578342    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rzvjc" podStartSLOduration=4.464298267 podStartE2EDuration="22.578328903s" totalImagesPullingTime="18.114030636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:39.578210653 +0000 UTC m=+22.748060522" watchObservedRunningTime="2026-09-06 10:30:39.578328903 +0000 UTC m=+22.748178819"
Sep 06 10:30:40.009541 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.009391    2573 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 06 10:30:40.325125 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.325003    2573 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-06T10:30:40.009537571Z","UUID":"d465b429-b15a-4c7e-8255-070a35edba68","Handler":null,"Name":"","Endpoint":""}
Sep 06 10:30:40.326510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.326489    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 06 10:30:40.326614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.326527    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 06 10:30:40.505899 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.505856    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq" event={"ID":"d614386c-c370-4e35-8055-1bddfbaf8242","Type":"ContainerStarted","Data":"7b633a1b067f1c83c0e364cca4005fa271b89821a87941bec5a48f846726a48a"}
Sep 06 10:30:40.507940 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.507905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bwsbz" event={"ID":"d052ce88-8d66-4bd1-beff-f9a0eb6c8ab5","Type":"ContainerStarted","Data":"2e368156e1ab1b4f1c732672bbbcf3ea1f2998b0e00ef0397a5e61face308a48"}
Sep 06 10:30:40.522577 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.522525    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bwsbz" podStartSLOduration=5.362146079 podStartE2EDuration="23.522507408s" totalImagesPullingTime="18.160361329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:40.522047447 +0000 UTC m=+23.691897319" watchObservedRunningTime="2026-09-06 10:30:40.522507408 +0000 UTC m=+23.692357280"
Sep 06 10:30:40.638257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.638109    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:40.638852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:40.638824    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:41.346845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:41.346814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:41.346845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:41.346841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:41.347071 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:41.346815    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:41.347071 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:41.346923    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:41.347071 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:41.347006    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:41.347187 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:41.347095    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:41.512887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:41.512773    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq" event={"ID":"d614386c-c370-4e35-8055-1bddfbaf8242","Type":"ContainerStarted","Data":"cded1152c996dc911597b389ef3545e47e6d5a410cc7d9bd51d1ca252bdd2fdf"}
Sep 06 10:30:41.516773 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:41.516732    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"16fc6dbb37e75069271dc64b0d30aa79c4815dd6b1a0b8280a57cf6513d6203b"}
Sep 06 10:30:42.519098 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:42.519073    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:43.347278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:43.347312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:43.347278    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:43.347412    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:43.347486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:43.348559 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:43.347615    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:44.257607 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:44.257575    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:44.258062 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:44.257693    2573 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 06 10:30:44.258455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:44.258436    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wsb9h"
Sep 06 10:30:44.272437 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:44.272363    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d2nlq" podStartSLOduration=7.034339217 podStartE2EDuration="27.272348232s" totalImagesPullingTime="20.238009015s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:41.529772068 +0000 UTC m=+24.699621941" watchObservedRunningTime="2026-09-06 10:30:44.272348232 +0000 UTC m=+27.442198118"
Sep 06 10:30:45.346803 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.346560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:45.346803 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.346560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:45.347358 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.346622    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:45.347358 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:45.346847    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:45.347358 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:45.346965    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:45.347358 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:45.347030    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:45.530961 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.530917    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" event={"ID":"b249010b-f2cd-4131-b7d3-0146ae3c3922","Type":"ContainerStarted","Data":"612435d9ac26cc3d2bc00d927d34dd3bf97de0ce11452ff079bac538dd897faa"}
Sep 06 10:30:45.531296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.531271    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:45.531408 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.531344    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:45.533129 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.533103    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="4563852762ea69e2dc87931be73deb8cb92fbb12f0a20a46bbb8732fd10c1447" exitCode=0
Sep 06 10:30:45.533262 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.533153    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"4563852762ea69e2dc87931be73deb8cb92fbb12f0a20a46bbb8732fd10c1447"}
Sep 06 10:30:45.549042 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.549008    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:45.556100 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:45.556048    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl" podStartSLOduration=10.379370075 podStartE2EDuration="28.556032051s" totalImagesPullingTime="18.176661976s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:45.555588328 +0000 UTC m=+28.725438199" watchObservedRunningTime="2026-09-06 10:30:45.556032051 +0000 UTC m=+28.725881973"
Sep 06 10:30:46.538179 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:46.538132    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="197ca3fa5dc66e7a0a284f5e5f6c2e225a01e53c40185d21048d9b495d3240e1" exitCode=0
Sep 06 10:30:46.538636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:46.538209    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"197ca3fa5dc66e7a0a284f5e5f6c2e225a01e53c40185d21048d9b495d3240e1"}
Sep 06 10:30:46.538711 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:46.538684    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:46.554906 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:46.554877    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:30:47.347027 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:47.346936    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:47.347183 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:47.347028    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:47.347183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:47.347044    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:47.347271 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:47.347183    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:47.347271 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:47.347219    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:47.347353 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:47.347331    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:47.543094 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:47.543051    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="605da4248757b1c63f297f8d4ce58f939b2c06e6a868ac23778916c372e37a62" exitCode=0
Sep 06 10:30:47.543554 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:47.543139    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"605da4248757b1c63f297f8d4ce58f939b2c06e6a868ac23778916c372e37a62"}
Sep 06 10:30:49.347403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:49.347359    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:49.348008 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:49.347359    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:49.348008 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:49.347513    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:49.348008 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:49.347526    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:49.348008 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:49.347614    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:49.348008 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:49.347690    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:51.114639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.114603    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:51.115108 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.114676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:51.115108 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.114789    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:51.115108 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.114857    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret podName:7508edae-7d98-4088-a13f-ac16080cdf40 nodeName:}" failed. No retries permitted until 2026-09-06 10:31:23.114841086 +0000 UTC m=+66.284690954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret") pod "global-pull-secret-syncer-n764h" (UID: "7508edae-7d98-4088-a13f-ac16080cdf40") : object "kube-system"/"original-pull-secret" not registered
Sep 06 10:30:51.115108 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.114789    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:51.115108 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.114954    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs podName:4e2f82a0-2b33-4585-b5a4-072fadb0f89d nodeName:}" failed. No retries permitted until 2026-09-06 10:31:23.114936439 +0000 UTC m=+66.284786294 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs") pod "network-metrics-daemon-dkvt2" (UID: "4e2f82a0-2b33-4585-b5a4-072fadb0f89d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 06 10:30:51.215929 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.215893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:51.216093 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.216078    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 06 10:30:51.216135 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.216098    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 06 10:30:51.216135 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.216108    2573 projected.go:196] Error preparing data for projected volume kube-api-access-jqvwd for pod openshift-network-diagnostics/network-check-target-hf4tq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:51.216291 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.216184    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd podName:4a14e7bf-6366-4c91-8377-5a1bd74e0c27 nodeName:}" failed. No retries permitted until 2026-09-06 10:31:23.216148939 +0000 UTC m=+66.385998801 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jqvwd" (UniqueName: "kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd") pod "network-check-target-hf4tq" (UID: "4a14e7bf-6366-4c91-8377-5a1bd74e0c27") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 06 10:30:51.346952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.346908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:51.347142 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.346908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:51.347142 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.347048    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:51.347142 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.347125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:51.347142 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:51.346908    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:51.347382 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:51.347229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:52.558987 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:52.558895    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="9d9f34cdef108f9ef7c858d5917ce94f3c7d043669440994fe10cd229212710c" exitCode=0
Sep 06 10:30:52.558987 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:52.558973    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"9d9f34cdef108f9ef7c858d5917ce94f3c7d043669440994fe10cd229212710c"}
Sep 06 10:30:53.347355 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:53.347320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:53.347579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:53.347320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:53.347579 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:53.347455    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:53.347579 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:53.347518    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:53.347579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:53.347566    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:53.347776 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:53.347625    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:53.565143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:53.565099    2573 generic.go:356] "Generic (PLEG): container finished" podID="631ae72d-88b6-4777-8e72-53906c337bb2" containerID="7ffb11082c8352e099b479539cd34116f252fc831a6d88d46cb30504c1d830a6" exitCode=0
Sep 06 10:30:53.565591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:53.565179    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerDied","Data":"7ffb11082c8352e099b479539cd34116f252fc831a6d88d46cb30504c1d830a6"}
Sep 06 10:30:54.571214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:54.570994    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kzp48" event={"ID":"631ae72d-88b6-4777-8e72-53906c337bb2","Type":"ContainerStarted","Data":"778efec9c1afcfe6b006d646bf0fb134265a58a8c98194fc05c2a003d430d9d8"}
Sep 06 10:30:54.591293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:54.591236    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kzp48" podStartSLOduration=7.988111971 podStartE2EDuration="37.591220565s" totalImagesPullingTime="29.603108594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:30:54.590219756 +0000 UTC m=+37.760069627" watchObservedRunningTime="2026-09-06 10:30:54.591220565 +0000 UTC m=+37.761070492"
Sep 06 10:30:55.346930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:55.346893    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:55.347106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:55.346906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:55.347106 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:55.347023    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:55.347106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:55.346906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:55.347106 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:55.347074    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:55.347373 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:55.347145    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:57.351358 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:57.351331    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:57.351797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:57.351416    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:57.351797 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:57.351450    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:57.351797 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:57.351486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:57.351797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:57.351524    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:57.351797 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:57.351578    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:30:58.904033 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.903993    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n764h"]
Sep 06 10:30:58.904574 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.904114    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:30:58.904574 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:58.904235    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:30:58.906013 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.905985    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hf4tq"]
Sep 06 10:30:58.906319 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.906290    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:30:58.906553 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:58.906529    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:30:58.906747 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.906723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkvt2"]
Sep 06 10:30:58.906822 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:30:58.906810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:30:58.906936 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:30:58.906914    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:31:00.346338 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:00.346295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:00.346338 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:00.346338    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:00.346764 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:00.346432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:31:00.346764 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:00.346444    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:31:01.347130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:01.347087    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:01.347577 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:01.347242    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:31:02.346885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:02.346850    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:02.347057 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:02.346977    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dkvt2" podUID="4e2f82a0-2b33-4585-b5a4-072fadb0f89d"
Sep 06 10:31:02.347057 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:02.347017    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:02.347214 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:02.347079    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hf4tq" podUID="4a14e7bf-6366-4c91-8377-5a1bd74e0c27"
Sep 06 10:31:03.347141 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.347050    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:03.347323 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:03.347194    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-n764h" podUID="7508edae-7d98-4088-a13f-ac16080cdf40"
Sep 06 10:31:03.664617 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.664591    2573 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-119.ec2.internal" event="NodeReady"
Sep 06 10:31:03.664770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.664719    2573 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 06 10:31:03.707229 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.707193    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-x9whf"]
Sep 06 10:31:03.725610 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.725579    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n5fvv"]
Sep 06 10:31:03.725782 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.725710    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.727994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.727916    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 06 10:31:03.728289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.728272    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-89vqh\""
Sep 06 10:31:03.728542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.728522    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 06 10:31:03.745770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.745008    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9whf"]
Sep 06 10:31:03.745770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.745315    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n5fvv"]
Sep 06 10:31:03.745770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.745097    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.745770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.745351    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bhl5s"]
Sep 06 10:31:03.747556 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.747532    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 06 10:31:03.747696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.747613    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 06 10:31:03.747696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.747534    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 06 10:31:03.747696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.747534    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l4ftj\""
Sep 06 10:31:03.747696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.747532    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 06 10:31:03.769000 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.768960    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhl5s"]
Sep 06 10:31:03.769181 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.769077    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:03.771013 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.770992    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 06 10:31:03.771151 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.771088    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 06 10:31:03.771291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.771271    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 06 10:31:03.771416 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.771351    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-66txg\""
Sep 06 10:31:03.805649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.805606    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a06dd0-f982-4bea-a575-147f2ea38801-tmp-dir\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.805798 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.805657    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a06dd0-f982-4bea-a575-147f2ea38801-config-volume\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.805798 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.805710    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a06dd0-f982-4bea-a575-147f2ea38801-metrics-tls\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.805798 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.805730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpbqt\" (UniqueName: \"kubernetes.io/projected/e2a06dd0-f982-4bea-a575-147f2ea38801-kube-api-access-hpbqt\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.902556 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.902520    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dx74m"]
Sep 06 10:31:03.907081 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpbqt\" (UniqueName: \"kubernetes.io/projected/e2a06dd0-f982-4bea-a575-147f2ea38801-kube-api-access-hpbqt\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.907191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmh9\" (UniqueName: \"kubernetes.io/projected/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-api-access-mmmh9\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.907191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907150    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a06dd0-f982-4bea-a575-147f2ea38801-tmp-dir\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.907260 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907207    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98dcb88b-0924-45a3-870b-8fae24b02936-cert\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:03.907378 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907358    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a06dd0-f982-4bea-a575-147f2ea38801-config-volume\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.907417 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907404    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-data-volume\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.907464 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907425    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.907533 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b2z6\" (UniqueName: \"kubernetes.io/projected/98dcb88b-0924-45a3-870b-8fae24b02936-kube-api-access-8b2z6\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:03.907590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-crio-socket\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.907590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:03.907688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.907627    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a06dd0-f982-4bea-a575-147f2ea38801-metrics-tls\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.910999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.908041    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2a06dd0-f982-4bea-a575-147f2ea38801-tmp-dir\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.910999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.908291    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a06dd0-f982-4bea-a575-147f2ea38801-config-volume\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.913484 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.913459    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2a06dd0-f982-4bea-a575-147f2ea38801-metrics-tls\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.916669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.916605    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpbqt\" (UniqueName: \"kubernetes.io/projected/e2a06dd0-f982-4bea-a575-147f2ea38801-kube-api-access-hpbqt\") pod \"dns-default-x9whf\" (UID: \"e2a06dd0-f982-4bea-a575-147f2ea38801\") " pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:03.932461 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.932422    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dx74m"]
Sep 06 10:31:03.932614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.932558    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:03.934990 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.934949    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 06 10:31:03.934990 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.934950    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 06 10:31:03.935224 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.934950    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 06 10:31:03.935224 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.935079    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-btxcd\""
Sep 06 10:31:03.935224 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.934955    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 06 10:31:03.935224 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:03.934955    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 06 10:31:04.008284 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008241    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98dcb88b-0924-45a3-870b-8fae24b02936-cert\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:04.008482 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28l8z\" (UniqueName: \"kubernetes.io/projected/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-kube-api-access-28l8z\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.008482 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008383    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.008482 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008455    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-data-volume\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008482 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008477    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.008692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008588    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.008692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008624    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b2z6\" (UniqueName: \"kubernetes.io/projected/98dcb88b-0924-45a3-870b-8fae24b02936-kube-api-access-8b2z6\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:04.008692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-crio-socket\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008950 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-data-volume\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008950 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008889    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmh9\" (UniqueName: \"kubernetes.io/projected/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-api-access-mmmh9\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.008950 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.008930    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-crio-socket\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.009096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.009078    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.011026 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.010993    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98dcb88b-0924-45a3-870b-8fae24b02936-cert\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:04.011138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.011066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.017090 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.017066    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmh9\" (UniqueName: \"kubernetes.io/projected/408d6e0d-907d-42d3-9364-9b2e37e3ed6d-kube-api-access-mmmh9\") pod \"insights-runtime-extractor-n5fvv\" (UID: \"408d6e0d-907d-42d3-9364-9b2e37e3ed6d\") " pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.017254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.017235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b2z6\" (UniqueName: \"kubernetes.io/projected/98dcb88b-0924-45a3-870b-8fae24b02936-kube-api-access-8b2z6\") pod \"ingress-canary-bhl5s\" (UID: \"98dcb88b-0924-45a3-870b-8fae24b02936\") " pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:04.037197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.037146    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:04.054342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.054294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n5fvv"
Sep 06 10:31:04.077885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.077862    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bhl5s"
Sep 06 10:31:04.109388 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.109362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28l8z\" (UniqueName: \"kubernetes.io/projected/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-kube-api-access-28l8z\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.109445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.109406    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.109490 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.109442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.109548 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.109489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.111754 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.110204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.112504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.112472    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.112603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.112481    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.119251 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.119228    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28l8z\" (UniqueName: \"kubernetes.io/projected/4ee43b66-46fd-4432-9d0d-fb7cfaf5243a-kube-api-access-28l8z\") pod \"prometheus-operator-6f4946db56-dx74m\" (UID: \"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.242006 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.241964    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m"
Sep 06 10:31:04.347068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.347016    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:04.347068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.347054    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:04.349924 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.349901    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:31:04.350586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.349911    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:31:04.350586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.349952    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bx9l2\""
Sep 06 10:31:04.350586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.350005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h4kvp\""
Sep 06 10:31:04.350586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.350098    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:31:04.385509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.385480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-x9whf"]
Sep 06 10:31:04.388672 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:04.388636    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a06dd0_f982_4bea_a575_147f2ea38801.slice/crio-360e771f583d3ddf00d2404244b9a80d53359382f7f1e46fe38e5487a5ca205c WatchSource:0}: Error finding container 360e771f583d3ddf00d2404244b9a80d53359382f7f1e46fe38e5487a5ca205c: Status 404 returned error can't find the container with id 360e771f583d3ddf00d2404244b9a80d53359382f7f1e46fe38e5487a5ca205c
Sep 06 10:31:04.391319 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.391244    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n5fvv"]
Sep 06 10:31:04.395193 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:04.395154    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408d6e0d_907d_42d3_9364_9b2e37e3ed6d.slice/crio-9a0c22f104e40be1dd192106abaedaf889f26573590322a181edfb5964fbde7b WatchSource:0}: Error finding container 9a0c22f104e40be1dd192106abaedaf889f26573590322a181edfb5964fbde7b: Status 404 returned error can't find the container with id 9a0c22f104e40be1dd192106abaedaf889f26573590322a181edfb5964fbde7b
Sep 06 10:31:04.421206 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.421153    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bhl5s"]
Sep 06 10:31:04.424921 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:04.424894    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98dcb88b_0924_45a3_870b_8fae24b02936.slice/crio-e3acabdd21dd39efde4ab0eff287d6fcf65af747f22b23d2596994ff0a3d11fe WatchSource:0}: Error finding container e3acabdd21dd39efde4ab0eff287d6fcf65af747f22b23d2596994ff0a3d11fe: Status 404 returned error can't find the container with id e3acabdd21dd39efde4ab0eff287d6fcf65af747f22b23d2596994ff0a3d11fe
Sep 06 10:31:04.529704 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.529669    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-dx74m"]
Sep 06 10:31:04.533112 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:04.533080    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee43b66_46fd_4432_9d0d_fb7cfaf5243a.slice/crio-faf7f6e203be073bd5c986fe08337723a0bccbba268132b008de8a372783ee33 WatchSource:0}: Error finding container faf7f6e203be073bd5c986fe08337723a0bccbba268132b008de8a372783ee33: Status 404 returned error can't find the container with id faf7f6e203be073bd5c986fe08337723a0bccbba268132b008de8a372783ee33
Sep 06 10:31:04.595721 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.595684    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhl5s" event={"ID":"98dcb88b-0924-45a3-870b-8fae24b02936","Type":"ContainerStarted","Data":"e3acabdd21dd39efde4ab0eff287d6fcf65af747f22b23d2596994ff0a3d11fe"}
Sep 06 10:31:04.596919 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.596884    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9whf" event={"ID":"e2a06dd0-f982-4bea-a575-147f2ea38801","Type":"ContainerStarted","Data":"360e771f583d3ddf00d2404244b9a80d53359382f7f1e46fe38e5487a5ca205c"}
Sep 06 10:31:04.598035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.597997    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m" event={"ID":"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a","Type":"ContainerStarted","Data":"faf7f6e203be073bd5c986fe08337723a0bccbba268132b008de8a372783ee33"}
Sep 06 10:31:04.599477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.599457    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5fvv" event={"ID":"408d6e0d-907d-42d3-9364-9b2e37e3ed6d","Type":"ContainerStarted","Data":"77f2bbe8bcccc4a6184fbc097ec6b90752cad17caf9a93bd92efa68f8080c94d"}
Sep 06 10:31:04.599584 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:04.599496    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5fvv" event={"ID":"408d6e0d-907d-42d3-9364-9b2e37e3ed6d","Type":"ContainerStarted","Data":"9a0c22f104e40be1dd192106abaedaf889f26573590322a181edfb5964fbde7b"}
Sep 06 10:31:05.346936 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:05.346547    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:05.350528 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:05.350323    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:31:06.607731 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:06.607477    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5fvv" event={"ID":"408d6e0d-907d-42d3-9364-9b2e37e3ed6d","Type":"ContainerStarted","Data":"6b94ff1cd4e27fb506fd81b73c00d9172862f28877d35acb85b4001959ac48b6"}
Sep 06 10:31:08.617611 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.617504    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m" event={"ID":"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a","Type":"ContainerStarted","Data":"1788f9678828213361e97b71805abfa70edf0d4f5fcafed913f0ef06da358d2a"}
Sep 06 10:31:08.618050 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.617614    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m" event={"ID":"4ee43b66-46fd-4432-9d0d-fb7cfaf5243a","Type":"ContainerStarted","Data":"6d191aeae94e5557f1480d136390da994868348eb381d0b01dce8c220eb104d0"}
Sep 06 10:31:08.619249 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.619202    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bhl5s" event={"ID":"98dcb88b-0924-45a3-870b-8fae24b02936","Type":"ContainerStarted","Data":"f046812bedfd6d41133f1106110cf28417855b6b060ab41299fe690409c6f799"}
Sep 06 10:31:08.621192 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.621147    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9whf" event={"ID":"e2a06dd0-f982-4bea-a575-147f2ea38801","Type":"ContainerStarted","Data":"b20051a836c004916f703726220d1998e565a22459b609c92e5bd11eba4817c2"}
Sep 06 10:31:08.621325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.621207    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-x9whf" event={"ID":"e2a06dd0-f982-4bea-a575-147f2ea38801","Type":"ContainerStarted","Data":"0bc331fe35ac7cd58c5f34d5067403874816f025a5af782035466e8035c4c8bd"}
Sep 06 10:31:08.621325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.621306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:08.639669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.639601    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-dx74m" podStartSLOduration=2.311159672 podStartE2EDuration="5.639578415s" totalImagesPullingTime="3.328418743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:08.63806346 +0000 UTC m=+51.807913333" watchObservedRunningTime="2026-09-06 10:31:08.639578415 +0000 UTC m=+51.809428285"
Sep 06 10:31:08.668054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.667990    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bhl5s" podStartSLOduration=2.226880901 podStartE2EDuration="5.667964544s" totalImagesPullingTime="3.441083643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:08.666458368 +0000 UTC m=+51.836308242" watchObservedRunningTime="2026-09-06 10:31:08.667964544 +0000 UTC m=+51.837814420"
Sep 06 10:31:08.693206 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:08.693133    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-x9whf" podStartSLOduration=2.220085892 podStartE2EDuration="5.693113608s" totalImagesPullingTime="3.473027716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:08.692295653 +0000 UTC m=+51.862145527" watchObservedRunningTime="2026-09-06 10:31:08.693113608 +0000 UTC m=+51.862963470"
Sep 06 10:31:09.627198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:09.626900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n5fvv" event={"ID":"408d6e0d-907d-42d3-9364-9b2e37e3ed6d","Type":"ContainerStarted","Data":"28b638026e41f828df371f888fb6b1462c31c99b84d1276d3d5667fa8fbba5cd"}
Sep 06 10:31:09.651197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:09.651065    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n5fvv" podStartSLOduration=2.712736535 podStartE2EDuration="6.651044463s" totalImagesPullingTime="3.938307928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:31:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:09.650906782 +0000 UTC m=+52.820756655" watchObservedRunningTime="2026-09-06 10:31:09.651044463 +0000 UTC m=+52.820894347"
Sep 06 10:31:10.348217 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.347959    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g2ddf"]
Sep 06 10:31:10.350405 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.350379    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"]
Sep 06 10:31:10.350564 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.350513    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.352338 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.352316    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.355440 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.355417    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 06 10:31:10.355740 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.355716    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 06 10:31:10.355992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.355950    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 06 10:31:10.356081 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.356038    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lw42m\""
Sep 06 10:31:10.356140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.356095    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 06 10:31:10.356140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.356098    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v6mwz\""
Sep 06 10:31:10.356140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.356038    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 06 10:31:10.360184 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.356849    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 06 10:31:10.384243 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.384202    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"]
Sep 06 10:31:10.463553 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463553 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463553    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/261b99cc-dc72-43d8-aa15-b3040591c16e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-sys\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463605    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463630    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8hsq\" (UniqueName: \"kubernetes.io/projected/cf1f9b21-a9c3-4068-a795-0f4114b70ced-kube-api-access-d8hsq\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463649    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463677    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-metrics-client-ca\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463713    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-textfile\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463738    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.463834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9twp\" (UniqueName: \"kubernetes.io/projected/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-api-access-v9twp\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.464197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463852    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.464197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-wtmp\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.464197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463913    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.464197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-root\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.464197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.463963    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-tls\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565283 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565245    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-textfile\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565283 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565292    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.565568 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9twp\" (UniqueName: \"kubernetes.io/projected/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-api-access-v9twp\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.565568 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.565568 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-wtmp\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565596    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.565698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565615    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-root\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565593    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-textfile\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565690    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-tls\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565737    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-wtmp\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565726    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-root\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565780    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.565825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565810    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/261b99cc-dc72-43d8-aa15-b3040591c16e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565843    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-sys\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565871    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565899    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8hsq\" (UniqueName: \"kubernetes.io/projected/cf1f9b21-a9c3-4068-a795-0f4114b70ced-kube-api-access-d8hsq\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565923    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565927    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf1f9b21-a9c3-4068-a795-0f4114b70ced-sys\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.565953    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-metrics-client-ca\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.566198    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/261b99cc-dc72-43d8-aa15-b3040591c16e-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.566370 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.566351    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.566414 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.566401    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.566570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.566535    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-metrics-client-ca\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.566695 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.566668    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.569989 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.569966    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.570096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.570081    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cf1f9b21-a9c3-4068-a795-0f4114b70ced-node-exporter-tls\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.570239 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.570218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.570282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.570267    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.573488 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.573463    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8hsq\" (UniqueName: \"kubernetes.io/projected/cf1f9b21-a9c3-4068-a795-0f4114b70ced-kube-api-access-d8hsq\") pod \"node-exporter-g2ddf\" (UID: \"cf1f9b21-a9c3-4068-a795-0f4114b70ced\") " pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.573774 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.573757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9twp\" (UniqueName: \"kubernetes.io/projected/261b99cc-dc72-43d8-aa15-b3040591c16e-kube-api-access-v9twp\") pod \"kube-state-metrics-74cb57df7-n2n2q\" (UID: \"261b99cc-dc72-43d8-aa15-b3040591c16e\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.664764 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.664728    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2ddf"
Sep 06 10:31:10.670820 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.670801    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"
Sep 06 10:31:10.677015 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:10.676984    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf1f9b21_a9c3_4068_a795_0f4114b70ced.slice/crio-6df612483099e61e4a1c6bca82ce92312c57d682958e431b7c9054318f165a8a WatchSource:0}: Error finding container 6df612483099e61e4a1c6bca82ce92312c57d682958e431b7c9054318f165a8a: Status 404 returned error can't find the container with id 6df612483099e61e4a1c6bca82ce92312c57d682958e431b7c9054318f165a8a
Sep 06 10:31:10.944692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:10.944360    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q"]
Sep 06 10:31:10.948584 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:10.948551    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261b99cc_dc72_43d8_aa15_b3040591c16e.slice/crio-a016b949c266c13f962329c08a02042c38aa9195c6a41badc2889fb189c01b24 WatchSource:0}: Error finding container a016b949c266c13f962329c08a02042c38aa9195c6a41badc2889fb189c01b24: Status 404 returned error can't find the container with id a016b949c266c13f962329c08a02042c38aa9195c6a41badc2889fb189c01b24
Sep 06 10:31:11.635271 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:11.635234    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q" event={"ID":"261b99cc-dc72-43d8-aa15-b3040591c16e","Type":"ContainerStarted","Data":"a016b949c266c13f962329c08a02042c38aa9195c6a41badc2889fb189c01b24"}
Sep 06 10:31:11.636900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:11.636867    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2ddf" event={"ID":"cf1f9b21-a9c3-4068-a795-0f4114b70ced","Type":"ContainerStarted","Data":"6df612483099e61e4a1c6bca82ce92312c57d682958e431b7c9054318f165a8a"}
Sep 06 10:31:12.641965 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.641934    2573 generic.go:356] "Generic (PLEG): container finished" podID="cf1f9b21-a9c3-4068-a795-0f4114b70ced" containerID="d1dd7e45979046086a2937e5e177c4e19e6bffa5844184900578bf6ccd3b6d7a" exitCode=0
Sep 06 10:31:12.642606 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.642018    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2ddf" event={"ID":"cf1f9b21-a9c3-4068-a795-0f4114b70ced","Type":"ContainerDied","Data":"d1dd7e45979046086a2937e5e177c4e19e6bffa5844184900578bf6ccd3b6d7a"}
Sep 06 10:31:12.644470 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.644433    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q" event={"ID":"261b99cc-dc72-43d8-aa15-b3040591c16e","Type":"ContainerStarted","Data":"4f9c56253f7a462701e16eab069c3050bf57e07e76738185740b8ab5155bde9f"}
Sep 06 10:31:12.644605 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.644487    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q" event={"ID":"261b99cc-dc72-43d8-aa15-b3040591c16e","Type":"ContainerStarted","Data":"7b50ff9ee0322e59ab9611a962fbd3c6b4069501496190be5d9791e57ed14e1f"}
Sep 06 10:31:12.644605 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.644522    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q" event={"ID":"261b99cc-dc72-43d8-aa15-b3040591c16e","Type":"ContainerStarted","Data":"15c85d4f9fa92d139f96a768579f63c59295663d63269f083b59cc7189cf35d4"}
Sep 06 10:31:12.674455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:12.674401    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-n2n2q" podStartSLOduration=1.443862939 podStartE2EDuration="2.674382694s" totalImagesPullingTime="1.230519755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:12.672800192 +0000 UTC m=+55.842650075" watchObservedRunningTime="2026-09-06 10:31:12.674382694 +0000 UTC m=+55.844232566"
Sep 06 10:31:13.651887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:13.651848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2ddf" event={"ID":"cf1f9b21-a9c3-4068-a795-0f4114b70ced","Type":"ContainerStarted","Data":"5d2419a549f1a3586dded7899f7231e1126f0a1da8f26f6f88868785b26aaa2b"}
Sep 06 10:31:13.652331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:13.651904    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2ddf" event={"ID":"cf1f9b21-a9c3-4068-a795-0f4114b70ced","Type":"ContainerStarted","Data":"c57606b3e024217c780d3ccd18a82b43edb2f7ba89efcef00259ebe98fd5befd"}
Sep 06 10:31:13.669346 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:13.669283    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g2ddf" podStartSLOduration=2.578505597 podStartE2EDuration="3.669267136s" totalImagesPullingTime="1.010846762s" totalInitContainerRuntime="79.914777ms" isStatefulPod=true podCreationTimestamp="2026-09-06 10:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:13.668256192 +0000 UTC m=+56.838106073" watchObservedRunningTime="2026-09-06 10:31:13.669267136 +0000 UTC m=+56.839117007"
Sep 06 10:31:14.682423 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.682389    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-99d9f67d-f79h6"]
Sep 06 10:31:14.686397 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.686368    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.689080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689048    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 06 10:31:14.689080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689067    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 06 10:31:14.689080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689045    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 06 10:31:14.689331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689049    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 06 10:31:14.689331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689054    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-98bb7h9feroo8\""
Sep 06 10:31:14.689331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.689045    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-lqk42\""
Sep 06 10:31:14.694990 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.694958    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-99d9f67d-f79h6"]
Sep 06 10:31:14.801495 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801461    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5013522-89bb-4749-a539-a0b4263c1a73-audit-log\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801672 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801532    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-client-certs\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801672 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801676    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmvn\" (UniqueName: \"kubernetes.io/projected/e5013522-89bb-4749-a539-a0b4263c1a73-kube-api-access-wgmvn\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-tls\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-client-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.801948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.801825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-metrics-server-audit-profiles\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903008 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.902956    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-client-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903008 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-metrics-server-audit-profiles\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903071    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5013522-89bb-4749-a539-a0b4263c1a73-audit-log\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903105    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-client-certs\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903158    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmvn\" (UniqueName: \"kubernetes.io/projected/e5013522-89bb-4749-a539-a0b4263c1a73-kube-api-access-wgmvn\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-tls\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e5013522-89bb-4749-a539-a0b4263c1a73-audit-log\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.903948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.903924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.904259 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.904235    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e5013522-89bb-4749-a539-a0b4263c1a73-metrics-server-audit-profiles\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.906785 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.906751    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-client-ca-bundle\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.906785 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.906759    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-client-certs\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.906914 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.906803    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e5013522-89bb-4749-a539-a0b4263c1a73-secret-metrics-server-tls\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.910892 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.910869    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmvn\" (UniqueName: \"kubernetes.io/projected/e5013522-89bb-4749-a539-a0b4263c1a73-kube-api-access-wgmvn\") pod \"metrics-server-99d9f67d-f79h6\" (UID: \"e5013522-89bb-4749-a539-a0b4263c1a73\") " pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:14.996297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:14.996211    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:15.033646 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.033599    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"]
Sep 06 10:31:15.037276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.037052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.040849 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.040825    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 06 10:31:15.040994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.040879    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-v7jv4\""
Sep 06 10:31:15.046637 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.046606    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"]
Sep 06 10:31:15.205647 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.205591    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.205647 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.205631    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj2cc\" (UniqueName: \"kubernetes.io/projected/abefd3c4-dc1e-41aa-b694-755fc8163d18-kube-api-access-sj2cc\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.275677 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.275592    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-99d9f67d-f79h6"]
Sep 06 10:31:15.279199 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:15.279153    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5013522_89bb_4749_a539_a0b4263c1a73.slice/crio-84527cbb3373135432c89d73709cdef3757feb4fe301019312a65da12bbbb113 WatchSource:0}: Error finding container 84527cbb3373135432c89d73709cdef3757feb4fe301019312a65da12bbbb113: Status 404 returned error can't find the container with id 84527cbb3373135432c89d73709cdef3757feb4fe301019312a65da12bbbb113
Sep 06 10:31:15.310095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.310057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.310318 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.310138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj2cc\" (UniqueName: \"kubernetes.io/projected/abefd3c4-dc1e-41aa-b694-755fc8163d18-kube-api-access-sj2cc\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.310318 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:15.310282    2573 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 06 10:31:15.310426 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:31:15.310364    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert podName:abefd3c4-dc1e-41aa-b694-755fc8163d18 nodeName:}" failed. No retries permitted until 2026-09-06 10:31:15.810340556 +0000 UTC m=+58.980190422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-q2spn" (UID: "abefd3c4-dc1e-41aa-b694-755fc8163d18") : secret "monitoring-plugin-cert" not found
Sep 06 10:31:15.324857 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.324820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj2cc\" (UniqueName: \"kubernetes.io/projected/abefd3c4-dc1e-41aa-b694-755fc8163d18-kube-api-access-sj2cc\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.505469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.503780    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"]
Sep 06 10:31:15.529607 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.529510    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"]
Sep 06 10:31:15.529751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.529659    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.532143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532113    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9qcgf\""
Sep 06 10:31:15.532317 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532207    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 06 10:31:15.532317 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532257    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 06 10:31:15.532427 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532356    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 06 10:31:15.532533 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532516    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 06 10:31:15.532882 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.532724    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 06 10:31:15.541266 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.541243    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 06 10:31:15.612380 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58z6\" (UniqueName: \"kubernetes.io/projected/7898c289-f50a-4cf7-8121-238838f172c9-kube-api-access-w58z6\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612380 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612388    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-federate-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612538    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-serving-certs-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.612797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.612644    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-metrics-client-ca\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.659308 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.659264    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6" event={"ID":"e5013522-89bb-4749-a539-a0b4263c1a73","Type":"ContainerStarted","Data":"84527cbb3373135432c89d73709cdef3757feb4fe301019312a65da12bbbb113"}
Sep 06 10:31:15.712977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.712944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-federate-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713038    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-serving-certs-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713127    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-metrics-client-ca\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713260    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w58z6\" (UniqueName: \"kubernetes.io/projected/7898c289-f50a-4cf7-8121-238838f172c9-kube-api-access-w58z6\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.713520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.713286    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.714291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.714220    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-serving-certs-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.714291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.714219    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-metrics-client-ca\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.714291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.714285    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.716196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.716154    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.716477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.716453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-telemeter-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.716477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.716470    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-federate-client-tls\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.716586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.716460    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7898c289-f50a-4cf7-8121-238838f172c9-secret-telemeter-client\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.722053 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.722024    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58z6\" (UniqueName: \"kubernetes.io/projected/7898c289-f50a-4cf7-8121-238838f172c9-kube-api-access-w58z6\") pod \"telemeter-client-55b7fcfdcf-6dcfd\" (UID: \"7898c289-f50a-4cf7-8121-238838f172c9\") " pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.814558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.814466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.817016 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.816985    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/abefd3c4-dc1e-41aa-b694-755fc8163d18-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-q2spn\" (UID: \"abefd3c4-dc1e-41aa-b694-755fc8163d18\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:15.842946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.842911    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"
Sep 06 10:31:15.947963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:15.947926    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:16.009878 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.009836    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-rs548"]
Sep 06 10:31:16.042536 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.042506    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rs548"]
Sep 06 10:31:16.042678 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.042624    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:16.044951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.044930    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jq2tr\""
Sep 06 10:31:16.045226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.045211    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 06 10:31:16.045556 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.045540    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 06 10:31:16.119615 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.119520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4m5f\" (UniqueName: \"kubernetes.io/projected/ffcbe0a7-c7aa-49f1-974a-290a0701164b-kube-api-access-z4m5f\") pod \"downloads-85d98c998f-rs548\" (UID: \"ffcbe0a7-c7aa-49f1-974a-290a0701164b\") " pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:16.137989 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.137956    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd"]
Sep 06 10:31:16.141488 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:16.141450    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7898c289_f50a_4cf7_8121_238838f172c9.slice/crio-fbf69f8b716af4d27c7985462d2dc3ae5a15b2e84f065450943f56a83297b80d WatchSource:0}: Error finding container fbf69f8b716af4d27c7985462d2dc3ae5a15b2e84f065450943f56a83297b80d: Status 404 returned error can't find the container with id fbf69f8b716af4d27c7985462d2dc3ae5a15b2e84f065450943f56a83297b80d
Sep 06 10:31:16.220030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.219980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4m5f\" (UniqueName: \"kubernetes.io/projected/ffcbe0a7-c7aa-49f1-974a-290a0701164b-kube-api-access-z4m5f\") pod \"downloads-85d98c998f-rs548\" (UID: \"ffcbe0a7-c7aa-49f1-974a-290a0701164b\") " pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:16.234059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.234030    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4m5f\" (UniqueName: \"kubernetes.io/projected/ffcbe0a7-c7aa-49f1-974a-290a0701164b-kube-api-access-z4m5f\") pod \"downloads-85d98c998f-rs548\" (UID: \"ffcbe0a7-c7aa-49f1-974a-290a0701164b\") " pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:16.249469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.249381    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"]
Sep 06 10:31:16.254584 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:16.254529    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabefd3c4_dc1e_41aa_b694_755fc8163d18.slice/crio-4a8047a1240f57550cf1ed8922d3884d29e3a5412f3eb589bd9a336f6b77a841 WatchSource:0}: Error finding container 4a8047a1240f57550cf1ed8922d3884d29e3a5412f3eb589bd9a336f6b77a841: Status 404 returned error can't find the container with id 4a8047a1240f57550cf1ed8922d3884d29e3a5412f3eb589bd9a336f6b77a841
Sep 06 10:31:16.355394 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.355358    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:16.502181 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.502129    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:31:16.520198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.517042    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.521024 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521001    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521246    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e418cv05r9r8b\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521126    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521502    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521640    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521664    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521749    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7str\""
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521948    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.521984    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.521994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522003    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522017    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522037    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522069    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522109    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522188    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522208    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522225    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522287    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdvbq\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522308    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.522523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.522362    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.525241 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525093    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525324    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525352    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525458    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525487    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525551    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525669    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 10:31:16.525794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.525701    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 10:31:16.541628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.527695    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 10:31:16.546209 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.544647    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:31:16.623278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623154    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623264    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdvbq\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623635 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623326    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623767 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.623955 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.623899    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624263 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624240    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624295    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624365    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624385    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624407    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624626    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.624717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.624708    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.626872 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.626847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.628051 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.628002    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.628774 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.628715    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.628915 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.628797    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.629507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.629485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.630404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.630384    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.631049 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.631028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.633590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.633567    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.633857 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.633774    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.633857 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.633824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdvbq\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.634596 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.634562    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.634836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.634746    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.634836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.634780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.634836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.634815    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.635019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.634839    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.636184 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.636143    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.636472 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.636453    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.652848 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.652810    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-rs548"]
Sep 06 10:31:16.665926 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.665877    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn" event={"ID":"abefd3c4-dc1e-41aa-b694-755fc8163d18","Type":"ContainerStarted","Data":"4a8047a1240f57550cf1ed8922d3884d29e3a5412f3eb589bd9a336f6b77a841"}
Sep 06 10:31:16.667424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.667396    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd" event={"ID":"7898c289-f50a-4cf7-8121-238838f172c9","Type":"ContainerStarted","Data":"fbf69f8b716af4d27c7985462d2dc3ae5a15b2e84f065450943f56a83297b80d"}
Sep 06 10:31:16.858414 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:16.858376    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:16.952526 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:16.952493    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcbe0a7_c7aa_49f1_974a_290a0701164b.slice/crio-26570384c73be4ad3fc2e4d4323cbbd51fcdf297e27168b41297dfe924947607 WatchSource:0}: Error finding container 26570384c73be4ad3fc2e4d4323cbbd51fcdf297e27168b41297dfe924947607: Status 404 returned error can't find the container with id 26570384c73be4ad3fc2e4d4323cbbd51fcdf297e27168b41297dfe924947607
Sep 06 10:31:17.241976 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:17.241689    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:31:17.674953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:17.674905    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6" event={"ID":"e5013522-89bb-4749-a539-a0b4263c1a73","Type":"ContainerStarted","Data":"067f94b43e505c912afd7455087394ad1bc4128d4498f0df44201ba658999c8c"}
Sep 06 10:31:17.680427 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:17.680389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"091f2984967dee227295bf462003121230bab69c13a9d2a2f0ddd3dd7c8542ed"}
Sep 06 10:31:17.682329 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:17.682232    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rs548" event={"ID":"ffcbe0a7-c7aa-49f1-974a-290a0701164b","Type":"ContainerStarted","Data":"26570384c73be4ad3fc2e4d4323cbbd51fcdf297e27168b41297dfe924947607"}
Sep 06 10:31:17.693467 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:17.692922    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6" podStartSLOduration=1.973552687 podStartE2EDuration="3.692900358s" totalImagesPullingTime="1.719347671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:17.692007764 +0000 UTC m=+60.861857654" watchObservedRunningTime="2026-09-06 10:31:17.692900358 +0000 UTC m=+60.862750231"
Sep 06 10:31:18.560795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:18.560762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nr7sl"
Sep 06 10:31:18.629516 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:18.629483    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-x9whf"
Sep 06 10:31:19.695351 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.695318    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn" event={"ID":"abefd3c4-dc1e-41aa-b694-755fc8163d18","Type":"ContainerStarted","Data":"bdda194fb21a5a7460c7489df21b66e9daca50219904bfdd3eb0a4d90752e2be"}
Sep 06 10:31:19.695807 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.695416    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:19.697444 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.697397    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469" exitCode=0
Sep 06 10:31:19.697575 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.697494    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"}
Sep 06 10:31:19.700376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.700344    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd" event={"ID":"7898c289-f50a-4cf7-8121-238838f172c9","Type":"ContainerStarted","Data":"b27fe08966974bdacfa79c03f66443f5f53164fe2ae96e5cd8890d84514e4361"}
Sep 06 10:31:19.700475 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.700398    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd" event={"ID":"7898c289-f50a-4cf7-8121-238838f172c9","Type":"ContainerStarted","Data":"63ccb523cb307aa03cd6ac354e9e6dcadeb145f9415b2269e07d4f894c42f53b"}
Sep 06 10:31:19.700475 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.700424    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd" event={"ID":"7898c289-f50a-4cf7-8121-238838f172c9","Type":"ContainerStarted","Data":"150e53938f914b3e399be1a09dffc867a513103817cfbb67a5f70a6eb2ed8d8a"}
Sep 06 10:31:19.701395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.701371    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn"
Sep 06 10:31:19.710887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.710829    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-q2spn" podStartSLOduration=1.898362952 podStartE2EDuration="4.710815179s" totalImagesPullingTime="2.812452227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:19.710389404 +0000 UTC m=+62.880239293" watchObservedRunningTime="2026-09-06 10:31:19.710815179 +0000 UTC m=+62.880665050"
Sep 06 10:31:19.773964 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:19.773895    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55b7fcfdcf-6dcfd" podStartSLOduration=1.843568661 podStartE2EDuration="4.773875147s" totalImagesPullingTime="2.930306486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:19.77347051 +0000 UTC m=+62.943320382" watchObservedRunningTime="2026-09-06 10:31:19.773875147 +0000 UTC m=+62.943725019"
Sep 06 10:31:23.199627 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.199587    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:23.200243 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.199661    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:23.202146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.202119    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 06 10:31:23.202146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.202142    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 06 10:31:23.212885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.212857    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7508edae-7d98-4088-a13f-ac16080cdf40-original-pull-secret\") pod \"global-pull-secret-syncer-n764h\" (UID: \"7508edae-7d98-4088-a13f-ac16080cdf40\") " pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:23.212885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.212871    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e2f82a0-2b33-4585-b5a4-072fadb0f89d-metrics-certs\") pod \"network-metrics-daemon-dkvt2\" (UID: \"4e2f82a0-2b33-4585-b5a4-072fadb0f89d\") " pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:23.259564 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.259527    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bx9l2\""
Sep 06 10:31:23.268462 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.268430    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dkvt2"
Sep 06 10:31:23.301249 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.300872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:23.303382 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.303359    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 06 10:31:23.313928 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.313900    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 06 10:31:23.325930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.325894    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvwd\" (UniqueName: \"kubernetes.io/projected/4a14e7bf-6366-4c91-8377-5a1bd74e0c27-kube-api-access-jqvwd\") pod \"network-check-target-hf4tq\" (UID: \"4a14e7bf-6366-4c91-8377-5a1bd74e0c27\") " pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:23.360636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.360598    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-n764h"
Sep 06 10:31:23.564284 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.564118    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-h4kvp\""
Sep 06 10:31:23.572399 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.572365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:23.832990 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.832899    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-n764h"]
Sep 06 10:31:23.836832 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.836801    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dkvt2"]
Sep 06 10:31:23.878321 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:23.878283    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hf4tq"]
Sep 06 10:31:23.913958 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:23.913912    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7508edae_7d98_4088_a13f_ac16080cdf40.slice/crio-d8ca89ed0a530879cca288171809bdfa20e1a6407451907393fac3946a48e898 WatchSource:0}: Error finding container d8ca89ed0a530879cca288171809bdfa20e1a6407451907393fac3946a48e898: Status 404 returned error can't find the container with id d8ca89ed0a530879cca288171809bdfa20e1a6407451907393fac3946a48e898
Sep 06 10:31:23.914749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:23.914705    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2f82a0_2b33_4585_b5a4_072fadb0f89d.slice/crio-56e414972653765d1124877c650213cb829b425394db9cd221712dd7829a8f6d WatchSource:0}: Error finding container 56e414972653765d1124877c650213cb829b425394db9cd221712dd7829a8f6d: Status 404 returned error can't find the container with id 56e414972653765d1124877c650213cb829b425394db9cd221712dd7829a8f6d
Sep 06 10:31:23.915349 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:31:23.915325    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a14e7bf_6366_4c91_8377_5a1bd74e0c27.slice/crio-a4826179f3be8387310ea60b34ce68fbb5b3e7479c19678236037b92e8523762 WatchSource:0}: Error finding container a4826179f3be8387310ea60b34ce68fbb5b3e7479c19678236037b92e8523762: Status 404 returned error can't find the container with id a4826179f3be8387310ea60b34ce68fbb5b3e7479c19678236037b92e8523762
Sep 06 10:31:24.728590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:24.728544    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n764h" event={"ID":"7508edae-7d98-4088-a13f-ac16080cdf40","Type":"ContainerStarted","Data":"d8ca89ed0a530879cca288171809bdfa20e1a6407451907393fac3946a48e898"}
Sep 06 10:31:24.730759 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:24.730683    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkvt2" event={"ID":"4e2f82a0-2b33-4585-b5a4-072fadb0f89d","Type":"ContainerStarted","Data":"56e414972653765d1124877c650213cb829b425394db9cd221712dd7829a8f6d"}
Sep 06 10:31:24.734323 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:24.734154    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"}
Sep 06 10:31:24.734323 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:24.734255    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"}
Sep 06 10:31:24.736533 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:24.736476    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hf4tq" event={"ID":"4a14e7bf-6366-4c91-8377-5a1bd74e0c27","Type":"ContainerStarted","Data":"a4826179f3be8387310ea60b34ce68fbb5b3e7479c19678236037b92e8523762"}
Sep 06 10:31:25.746247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:25.745453    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkvt2" event={"ID":"4e2f82a0-2b33-4585-b5a4-072fadb0f89d","Type":"ContainerStarted","Data":"6cf8c4aadec0a2256457837b5144b1c515038bf6fc0d89e80bd44d0791c2bedc"}
Sep 06 10:31:25.746247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:25.745528    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dkvt2" event={"ID":"4e2f82a0-2b33-4585-b5a4-072fadb0f89d","Type":"ContainerStarted","Data":"7a02fdb1e052f7a0b86d40644e6fdec0fb05736a7cc0399b8163f105e7449f98"}
Sep 06 10:31:25.761373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:25.761120    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dkvt2" podStartSLOduration=67.5251631 podStartE2EDuration="1m8.761096403s" totalImagesPullingTime="1.235933303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:25.760628997 +0000 UTC m=+68.930478870" watchObservedRunningTime="2026-09-06 10:31:25.761096403 +0000 UTC m=+68.930946276"
Sep 06 10:31:34.996913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:34.996872    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:34.997549 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:34.996956    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:37.800080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.800030    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-n764h" event={"ID":"7508edae-7d98-4088-a13f-ac16080cdf40","Type":"ContainerStarted","Data":"0e8d5a5a6722be03761c592616eb8a9337a2ac76f965d26c366d94118bcf6050"}
Sep 06 10:31:37.805787 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.805749    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"}
Sep 06 10:31:37.806065 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.806036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"}
Sep 06 10:31:37.806210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.806091    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"}
Sep 06 10:31:37.806210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.806132    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerStarted","Data":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"}
Sep 06 10:31:37.808323 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.808296    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-rs548" event={"ID":"ffcbe0a7-c7aa-49f1-974a-290a0701164b","Type":"ContainerStarted","Data":"6fd627a7777a6ccec959d0b2160f8c1c672c17d0cb792765be3970598bd3c324"}
Sep 06 10:31:37.808457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.808434    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:37.810426 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.810401    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hf4tq" event={"ID":"4a14e7bf-6366-4c91-8377-5a1bd74e0c27","Type":"ContainerStarted","Data":"8a4a3d7df0dfe239f42e2d5ea4bf0fe9fd1281e45553e1b2ec1bccf4f414bf61"}
Sep 06 10:31:37.810889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.810867    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:31:37.814900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.814833    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-n764h" podStartSLOduration=67.843836479 podStartE2EDuration="1m20.814775159s" totalImagesPullingTime="12.97093868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:37.814345979 +0000 UTC m=+80.984195857" watchObservedRunningTime="2026-09-06 10:31:37.814775159 +0000 UTC m=+80.984625082"
Sep 06 10:31:37.823141 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.823112    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-rs548"
Sep 06 10:31:37.850990 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.850928    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-rs548" podStartSLOduration=2.664371753 podStartE2EDuration="22.850908604s" totalImagesPullingTime="20.186536851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:37.834315531 +0000 UTC m=+81.004165405" watchObservedRunningTime="2026-09-06 10:31:37.850908604 +0000 UTC m=+81.020758489"
Sep 06 10:31:37.851204 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.851077    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hf4tq" podStartSLOduration=67.891871984 podStartE2EDuration="1m20.851064998s" totalImagesPullingTime="12.959193014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:30:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:37.849349901 +0000 UTC m=+81.019199773" watchObservedRunningTime="2026-09-06 10:31:37.851064998 +0000 UTC m=+81.020914933"
Sep 06 10:31:37.877278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:37.877213    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.070923339 podStartE2EDuration="21.87719238s" totalImagesPullingTime="18.773320439s" totalInitContainerRuntime="32.948602ms" isStatefulPod=false podCreationTimestamp="2026-09-06 10:31:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:31:37.876123068 +0000 UTC m=+81.045972937" watchObservedRunningTime="2026-09-06 10:31:37.87719238 +0000 UTC m=+81.047042253"
Sep 06 10:31:41.859323 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:41.859279    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:54.713721 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:54.713679    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:54.770848 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:54.770814    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:54.903295 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:54.903266    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:31:55.004529 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:55.004454    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:31:55.008377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:31:55.008352    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-99d9f67d-f79h6"
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.843703    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844059    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="prometheus" containerID="cri-o://34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f" gracePeriod=600
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844080    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-web" containerID="cri-o://e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f" gracePeriod=600
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844107    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="thanos-sidecar" containerID="cri-o://65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52" gracePeriod=600
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844153    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy" containerID="cri-o://490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5" gracePeriod=600
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844224    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="config-reloader" containerID="cri-o://f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d" gracePeriod=600
Sep 06 10:32:04.844302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:04.844107    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-thanos" containerID="cri-o://f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d" gracePeriod=600
Sep 06 10:32:04.888763 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:04.888588    2573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 06 10:32:04.889906 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:04.889861    2573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 06 10:32:04.891022 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:04.890955    2573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Sep 06 10:32:04.891150 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:04.891029    2573 prober.go:104] "Probe errored" err="rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="prometheus" probeResult="unknown"
Sep 06 10:32:05.148285 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.147990    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:05.196375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196336    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196378    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196406    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196428    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196455    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196537    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196571    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196607    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196639    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196662    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdvbq\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196704    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196732    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196786    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196814    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196840    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196872    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196907    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.196951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196934    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.197473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196960    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls\") pod \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\" (UID: \"7d2cf942-a83e-48fb-bae4-8afeb21b51f5\") "
Sep 06 10:32:05.197473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.196965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.197473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.197317    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.197473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.197381    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.198085 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.198051    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.198793 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.198759    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.198900 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.198809    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.199983 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.199945    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:32:05.200432 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.200402    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.202835 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.202804    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:32:05.208613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208296    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.208613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208441    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.208613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208463    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq" (OuterVolumeSpecName: "kube-api-access-qdvbq") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "kube-api-access-qdvbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:32:05.208613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208541    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:32:05.208613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208592    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.208963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208651    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.208963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208769    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.208963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.208778    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out" (OuterVolumeSpecName: "config-out") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:32:05.209459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.209366    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.209459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.209411    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config" (OuterVolumeSpecName: "config") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.209633 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.209512    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.220045 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.220003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config" (OuterVolumeSpecName: "web-config") pod "7d2cf942-a83e-48fb-bae4-8afeb21b51f5" (UID: "7d2cf942-a83e-48fb-bae4-8afeb21b51f5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298567    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298639    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298654    2573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298667    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdvbq\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-kube-api-access-qdvbq\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298680    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298693    2573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298706    2573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-tls-assets\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298718    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298730    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298742    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-db\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298755    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298767    2573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298779    2573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-grpc-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298793    2573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-kube-rbac-proxy\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298804    2573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298819    2573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-secret-metrics-client-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298832    2573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-configmap-metrics-client-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298844    2573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-web-config\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.298885 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.298856    2573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7d2cf942-a83e-48fb-bae4-8afeb21b51f5-config-out\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:32:05.934788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934749    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d" exitCode=0
Sep 06 10:32:05.934788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934780    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5" exitCode=0
Sep 06 10:32:05.934788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934790    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f" exitCode=0
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934801    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52" exitCode=0
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934808    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d" exitCode=0
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934819    2573 generic.go:356] "Generic (PLEG): container finished" podID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f" exitCode=0
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934826    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934840    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934903    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934930    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934974    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.934980    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.935081    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"}
Sep 06 10:32:05.935404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.935106    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7d2cf942-a83e-48fb-bae4-8afeb21b51f5","Type":"ContainerDied","Data":"091f2984967dee227295bf462003121230bab69c13a9d2a2f0ddd3dd7c8542ed"}
Sep 06 10:32:05.945803 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.945714    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:05.954253 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.954122    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:05.957035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.956973    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:05.961364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.961335    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:05.963870 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.963846    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:05.973033 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.973009    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:05.990573 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.990519    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:05.992340 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992310    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:05.992773 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992755    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="prometheus"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992777    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="prometheus"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992796    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="config-reloader"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992804    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="config-reloader"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992817    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-web"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992828    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-web"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992845    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="init-config-reloader"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992854    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="init-config-reloader"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992868    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-thanos"
Sep 06 10:32:05.992877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992876    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-thanos"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992905    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="thanos-sidecar"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992914    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="thanos-sidecar"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992924    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.992933    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993065    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="prometheus"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993085    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="thanos-sidecar"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993095    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="config-reloader"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993105    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-web"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993114    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy"
Sep 06 10:32:05.993309 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:05.993129    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" containerName="kube-rbac-proxy-thanos"
Sep 06 10:32:06.003611 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.003574    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.015246 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.015215    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.015687 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.015667    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.015734 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.015698    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.015781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.015739    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.016005 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.015987    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.016075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016008    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.016075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016021    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.016337 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.016321    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.016384 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016354    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.016384 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016367    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.016609 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.016593    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.016659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016611    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.016659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016624    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.016872 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.016858    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.016911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016873    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.016911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.016884    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.017131 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.017113    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.017209 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017133    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.017209 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017146    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.017442 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:32:06.017414    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.017519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017451    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.017519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017466    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.018096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017733    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.018096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.017753    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.018096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018014    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.018096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018036    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.018395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018368    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.018475 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018397    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.019188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018714    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.019188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018735    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.019188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018943    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.019188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.018956    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.019455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019215    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.019455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019234    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.019520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019472    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.019520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019487    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.019687 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019672    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.019730 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019686    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.019891 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019872    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.019930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.019893    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.020155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020140    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.020216 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020157    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.020457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020438    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.020524 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020459    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.020731 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020708    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.020788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020731    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.020994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020977    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.021045 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.020994    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.021225 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021207    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.021225 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021224    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.021517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021498    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.021570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021517    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.021780 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021758    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.021829 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.021783    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.022071 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022049    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.022071 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022070    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.022379 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022358    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.022468 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022380    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.022689 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022661    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.022689 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022689    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.022985 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022966    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.022985 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.022985    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.023261 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023240    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.023304 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023262    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.023569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023544    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.023650 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023572    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.023818 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023795    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.023818 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.023817    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.024029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024012    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.024029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024028    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.024435 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024403    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.024435 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024421    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.024601 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024581    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.024601 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024599    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.024797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024780    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.024843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.024797    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.025068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025050    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.025138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025072    2573 scope.go:117] "RemoveContainer" containerID="f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"
Sep 06 10:32:06.025367 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025341    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d"} err="failed to get container status \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": rpc error: code = NotFound desc = could not find container \"f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d\": container with ID starting with f4a1ce866ef6ba371aa9fcb28d7ceb6b86ebaa31f342b620e264be911ab09e5d not found: ID does not exist"
Sep 06 10:32:06.025454 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025368    2573 scope.go:117] "RemoveContainer" containerID="490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"
Sep 06 10:32:06.025641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025623    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5"} err="failed to get container status \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": rpc error: code = NotFound desc = could not find container \"490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5\": container with ID starting with 490291563cc42db51a0a54e7bfa511e1647b8b6555c87e72db4c5ed017e4ece5 not found: ID does not exist"
Sep 06 10:32:06.025704 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025641    2573 scope.go:117] "RemoveContainer" containerID="e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"
Sep 06 10:32:06.025878 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025860    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f"} err="failed to get container status \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": rpc error: code = NotFound desc = could not find container \"e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f\": container with ID starting with e1637725801587950343b67584d78240de0e005ded4b6f47723fb08526741d2f not found: ID does not exist"
Sep 06 10:32:06.025922 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.025880    2573 scope.go:117] "RemoveContainer" containerID="65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"
Sep 06 10:32:06.026694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.026676    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52"} err="failed to get container status \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": rpc error: code = NotFound desc = could not find container \"65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52\": container with ID starting with 65278b09cb6422f87ccd2708a1f2637c6f98d951527d5728df55832242538e52 not found: ID does not exist"
Sep 06 10:32:06.026694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.026692    2573 scope.go:117] "RemoveContainer" containerID="f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"
Sep 06 10:32:06.026924 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.026902    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d"} err="failed to get container status \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": rpc error: code = NotFound desc = could not find container \"f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d\": container with ID starting with f3d8397e6981419a9364540b512672e86ebfb90f8c0ee3a1acad2fba6d2ec30d not found: ID does not exist"
Sep 06 10:32:06.026984 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.026926    2573 scope.go:117] "RemoveContainer" containerID="34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"
Sep 06 10:32:06.027198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.027177    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f"} err="failed to get container status \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": rpc error: code = NotFound desc = could not find container \"34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f\": container with ID starting with 34ac6bae79d8e8c55464f7a8d02cd679b9e96f23d237adde06345eb2d48a273f not found: ID does not exist"
Sep 06 10:32:06.027267 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.027200    2573 scope.go:117] "RemoveContainer" containerID="15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"
Sep 06 10:32:06.027455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.027436    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469"} err="failed to get container status \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": rpc error: code = NotFound desc = could not find container \"15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469\": container with ID starting with 15a695206cbca73f827af443cda156c4a1105433d03953796903f96008cc3469 not found: ID does not exist"
Sep 06 10:32:06.033929 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.033879    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:06.034069 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.034009    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.037269    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.037524    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.037707    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.037873    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.038077    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.038851    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.039050    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.039266    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e418cv05r9r8b\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.039485    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.039672    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-l7str\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.039892    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.040084    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.040723    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.047387    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 06 10:32:06.052254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.048460    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 06 10:32:06.053247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.053120    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 06 10:32:06.107385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107353    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107392    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107588 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107421    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107588 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107588 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107556    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107612    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107636    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107673    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107688    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7fzx\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-kube-api-access-l7fzx\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107753    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107800    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107823    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107855    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107920    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107936    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.107971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.108076 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.107975    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208407 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208407 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208623 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208434    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208623 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208707 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208634    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208707 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208666    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208705    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208738    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208763    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208793    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208799    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208815    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7fzx\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-kube-api-access-l7fzx\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208890    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208925    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.208959    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.208992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209043    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209190    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209357    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209428    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.209726 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.209702    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.210413 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.210385    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.210582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.210501    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.212191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.211934    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.212308 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.212189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.213352 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.212824    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-web-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.213352 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.213231    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.213991 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.213659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.213991 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.213693    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.214210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.214191    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.214394 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.214374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.215709 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.215669    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.216033 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.216007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-config-out\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.216304 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.216234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.216426 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.216404    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.216538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.216522    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.217796 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.217766    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7fzx\" (UniqueName: \"kubernetes.io/projected/8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4-kube-api-access-l7fzx\") pod \"prometheus-k8s-0\" (UID: \"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.352890 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.352845    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:32:06.674680 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.674643    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 06 10:32:06.678246 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:32:06.678214    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2a8237_b7c3_4a1f_aac0_f9ff3d4dbff4.slice/crio-926553211d23b35d61ee6671078fe4d5bac77cf028313a2ceca453e758bb77e0 WatchSource:0}: Error finding container 926553211d23b35d61ee6671078fe4d5bac77cf028313a2ceca453e758bb77e0: Status 404 returned error can't find the container with id 926553211d23b35d61ee6671078fe4d5bac77cf028313a2ceca453e758bb77e0
Sep 06 10:32:06.941942 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.941860    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4" containerID="269b59294211e8094835a07b7cf4e8e1af324c7c7a61059967d4c187a239bddf" exitCode=0
Sep 06 10:32:06.942321 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.941944    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerDied","Data":"269b59294211e8094835a07b7cf4e8e1af324c7c7a61059967d4c187a239bddf"}
Sep 06 10:32:06.942321 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:06.942011    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"926553211d23b35d61ee6671078fe4d5bac77cf028313a2ceca453e758bb77e0"}
Sep 06 10:32:07.366331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.366293    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d2cf942-a83e-48fb-bae4-8afeb21b51f5" path="/var/lib/kubelet/pods/7d2cf942-a83e-48fb-bae4-8afeb21b51f5/volumes"
Sep 06 10:32:07.949993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.949954    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"47934c60f5d27a25af8d936e397f1e9abe2a98b5fa036038a40a6e33ab0658c1"}
Sep 06 10:32:07.950403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.950016    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"23e5c0cf907fa7dd7fe4f20efc41162ac07f5d7e2ab5ab4e4c19cd9105b7970c"}
Sep 06 10:32:07.950403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.950042    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"e4a1f5cea0f8500b0ac49a7cf78d4925bf60b7e59d9048c4bbbe77fd622cfae9"}
Sep 06 10:32:07.950403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.950074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"a85d848049cfb0c4b46b29765f8255102d529df296710e13992191a2c856518a"}
Sep 06 10:32:07.950403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.950098    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"a8b03047d4c5948ef49fb528af0d39fe43daeea7a6ddeb28592f72302735bbac"}
Sep 06 10:32:07.950403 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.950122    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4","Type":"ContainerStarted","Data":"1ec59bea790d5f00ad0edcd0f31d292b7cba8ab3c8af3f475486817b1f05cd88"}
Sep 06 10:32:07.974720 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:07.974668    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.940049912 podStartE2EDuration="2.974652135s" totalImagesPullingTime="0s" totalInitContainerRuntime="34.602223ms" isStatefulPod=false podCreationTimestamp="2026-09-06 10:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:32:07.973985528 +0000 UTC m=+111.143835400" watchObservedRunningTime="2026-09-06 10:32:07.974652135 +0000 UTC m=+111.144502048"
Sep 06 10:32:09.823638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:09.823599    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hf4tq"
Sep 06 10:32:11.353058 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:32:11.353024    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:33:06.353673 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:06.353630    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:33:06.369685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:06.369660    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:33:07.191072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:07.191042    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 06 10:33:42.737455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.737405    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"]
Sep 06 10:33:42.741584 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.741560    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.743665 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.743646    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 06 10:33:42.743768 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.743654    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 06 10:33:42.744379 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.744359    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gfrht\""
Sep 06 10:33:42.747330 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.747310    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"]
Sep 06 10:33:42.797178 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.797135    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6dd4\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-kube-api-access-r6dd4\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.797314 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.797211    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.898469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.898433    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6dd4\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-kube-api-access-r6dd4\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.898610 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.898496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.906504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.906478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:42.906609 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:42.906534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6dd4\" (UniqueName: \"kubernetes.io/projected/7bfcda44-969e-443c-a714-63613b746f8f-kube-api-access-r6dd4\") pod \"cert-manager-cainjector-559cdd8c87-n52kb\" (UID: \"7bfcda44-969e-443c-a714-63613b746f8f\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:43.064714 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:43.064626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"
Sep 06 10:33:43.332698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:43.332626    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-n52kb"]
Sep 06 10:33:43.335606 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:33:43.335559    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bfcda44_969e_443c_a714_63613b746f8f.slice/crio-14383de78ee4a7bdab42796d4a61ccb533eb1c16cc15891213ab98d584aaf0cd WatchSource:0}: Error finding container 14383de78ee4a7bdab42796d4a61ccb533eb1c16cc15891213ab98d584aaf0cd: Status 404 returned error can't find the container with id 14383de78ee4a7bdab42796d4a61ccb533eb1c16cc15891213ab98d584aaf0cd
Sep 06 10:33:44.314341 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:44.314305    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb" event={"ID":"7bfcda44-969e-443c-a714-63613b746f8f","Type":"ContainerStarted","Data":"14383de78ee4a7bdab42796d4a61ccb533eb1c16cc15891213ab98d584aaf0cd"}
Sep 06 10:33:47.326701 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:47.326661    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb" event={"ID":"7bfcda44-969e-443c-a714-63613b746f8f","Type":"ContainerStarted","Data":"e76aa1465125df4825437d601000e7fdc7f911c2610ade0612c945d9168b1cd1"}
Sep 06 10:33:47.342444 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:33:47.342387    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-n52kb" podStartSLOduration=2.403491974 podStartE2EDuration="5.342364638s" totalImagesPullingTime="2.938872664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:33:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:33:47.341215774 +0000 UTC m=+210.511065667" watchObservedRunningTime="2026-09-06 10:33:47.342364638 +0000 UTC m=+210.512214512"
Sep 06 10:34:10.880541 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.880506    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"]
Sep 06 10:34:10.883770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.883755    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:10.887030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.887005    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 06 10:34:10.887592 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.887560    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 06 10:34:10.887696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.887642    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 06 10:34:10.887908 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.887893    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fg8xs\""
Sep 06 10:34:10.890570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.890555    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 06 10:34:10.913496 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:10.913471    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"]
Sep 06 10:34:11.037039 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.037002    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.037039 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.037043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-tmp\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.037529 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.037071    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-manifests\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.037529 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.037128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.037529 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.037228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cccv2\" (UniqueName: \"kubernetes.io/projected/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-kube-api-access-cccv2\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.138854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.138752    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.138854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.138798    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-tmp\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.138854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.138835    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-manifests\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.139130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.138861    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.139130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.138896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cccv2\" (UniqueName: \"kubernetes.io/projected/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-kube-api-access-cccv2\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.139340 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.139313    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-manifests\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.141466 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.141438    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-tmp\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.141553 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.141518    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.141593 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.141569    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-webhook-cert\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.149036 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.149017    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cccv2\" (UniqueName: \"kubernetes.io/projected/a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9-kube-api-access-cccv2\") pod \"opendatahub-operator-controller-manager-645f664568-z9k79\" (UID: \"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9\") " pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.194236 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.194204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:11.470831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:11.470805    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"]
Sep 06 10:34:11.473613 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:34:11.473585    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a719c3_ee83_406e_a3d1_4f69a8eb9cc9.slice/crio-6231d1a3b200bf47aafacc0e6dcd1c57a5be8705179ef4c3f4418592c9944527 WatchSource:0}: Error finding container 6231d1a3b200bf47aafacc0e6dcd1c57a5be8705179ef4c3f4418592c9944527: Status 404 returned error can't find the container with id 6231d1a3b200bf47aafacc0e6dcd1c57a5be8705179ef4c3f4418592c9944527
Sep 06 10:34:12.417918 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:12.417754    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79" event={"ID":"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9","Type":"ContainerStarted","Data":"6231d1a3b200bf47aafacc0e6dcd1c57a5be8705179ef4c3f4418592c9944527"}
Sep 06 10:34:15.431150 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:15.431112    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79" event={"ID":"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9","Type":"ContainerStarted","Data":"2bcfb9a2592223bcae80e17501e4a7a00be4683b41d4ef37eb469006facf608a"}
Sep 06 10:34:16.436869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:16.436835    2573 generic.go:356] "Generic (PLEG): container finished" podID="a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9" containerID="2bcfb9a2592223bcae80e17501e4a7a00be4683b41d4ef37eb469006facf608a" exitCode=0
Sep 06 10:34:16.437289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:16.436935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79" event={"ID":"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9","Type":"ContainerDied","Data":"2bcfb9a2592223bcae80e17501e4a7a00be4683b41d4ef37eb469006facf608a"}
Sep 06 10:34:17.446820 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:17.446786    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79" event={"ID":"a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9","Type":"ContainerStarted","Data":"61dc91cc7e9753d42c26804a9557f9f44081911d9a85fa5dcdd0ee216568a348"}
Sep 06 10:34:17.447201 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:17.446918    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:17.467549 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:17.467509    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79" podStartSLOduration=3.103431086 podStartE2EDuration="7.467496031s" totalImagesPullingTime="3.337511533s" totalInitContainerRuntime="1.026553412s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:17.465961039 +0000 UTC m=+240.635810910" watchObservedRunningTime="2026-09-06 10:34:17.467496031 +0000 UTC m=+240.637345902"
Sep 06 10:34:28.453507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:28.453472    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-645f664568-z9k79"
Sep 06 10:34:30.395685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.395650    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"]
Sep 06 10:34:30.404249 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.404224    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.411130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.411113    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 06 10:34:30.411474 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.411457    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 06 10:34:30.411952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.411933    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 06 10:34:30.412761 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.412743    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-p6lgl\""
Sep 06 10:34:30.413468 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.413453    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 06 10:34:30.415046 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.415031    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 06 10:34:30.420968 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.420946    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"]
Sep 06 10:34:30.502367 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.502334    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8vh\" (UniqueName: \"kubernetes.io/projected/2a0e893c-665f-490d-abfe-c3de0f5cee53-kube-api-access-sq8vh\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.502538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.502375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-metrics-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.502538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.502394    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2a0e893c-665f-490d-abfe-c3de0f5cee53-manager-config\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.502538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.502514    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.603619 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.603578    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8vh\" (UniqueName: \"kubernetes.io/projected/2a0e893c-665f-490d-abfe-c3de0f5cee53-kube-api-access-sq8vh\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.603733 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.603632    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-metrics-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.603733 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.603649    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2a0e893c-665f-490d-abfe-c3de0f5cee53-manager-config\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.603733 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.603699    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.604372 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.604345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/2a0e893c-665f-490d-abfe-c3de0f5cee53-manager-config\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.606148 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.606125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.606341 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.606319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/2a0e893c-665f-490d-abfe-c3de0f5cee53-metrics-cert\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.612363 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.612345    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8vh\" (UniqueName: \"kubernetes.io/projected/2a0e893c-665f-490d-abfe-c3de0f5cee53-kube-api-access-sq8vh\") pod \"lws-controller-manager-59b7bf7857-pzkbz\" (UID: \"2a0e893c-665f-490d-abfe-c3de0f5cee53\") " pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.713841 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.713757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:30.994737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:30.994709    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"]
Sep 06 10:34:30.997328 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:34:30.997301    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0e893c_665f_490d_abfe_c3de0f5cee53.slice/crio-d0b9f3600b928617ff2d88f115a34008a35b5766a6efddbdbf93fd9708006fcc WatchSource:0}: Error finding container d0b9f3600b928617ff2d88f115a34008a35b5766a6efddbdbf93fd9708006fcc: Status 404 returned error can't find the container with id d0b9f3600b928617ff2d88f115a34008a35b5766a6efddbdbf93fd9708006fcc
Sep 06 10:34:31.508558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:31.508517    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz" event={"ID":"2a0e893c-665f-490d-abfe-c3de0f5cee53","Type":"ContainerStarted","Data":"d0b9f3600b928617ff2d88f115a34008a35b5766a6efddbdbf93fd9708006fcc"}
Sep 06 10:34:34.522373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:34.522337    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz" event={"ID":"2a0e893c-665f-490d-abfe-c3de0f5cee53","Type":"ContainerStarted","Data":"0fda0888521f11a92cf242b4d832a9d7d3608a07a2d6093979f193fdabb001ed"}
Sep 06 10:34:34.522793 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:34.522448    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:34.538874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:34.538825    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz" podStartSLOduration=1.997836767 podStartE2EDuration="4.538811993s" totalImagesPullingTime="2.540975226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:34.537589682 +0000 UTC m=+257.707439555" watchObservedRunningTime="2026-09-06 10:34:34.538811993 +0000 UTC m=+257.708661864"
Sep 06 10:34:35.468878 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.468843    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"]
Sep 06 10:34:35.472115 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.472099    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.475091 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.475066    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 06 10:34:35.475199 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.475107    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 06 10:34:35.475736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.475715    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5qnct\""
Sep 06 10:34:35.476289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.476272    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 06 10:34:35.476528 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.476512    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 06 10:34:35.485437 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.485418    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"]
Sep 06 10:34:35.548451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.548417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bwjq\" (UniqueName: \"kubernetes.io/projected/516a8da8-1848-4b52-a302-e3d06e03d8f7-kube-api-access-4bwjq\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.548862 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.548484    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516a8da8-1848-4b52-a302-e3d06e03d8f7-tls-certs\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.548862 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.548533    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/516a8da8-1848-4b52-a302-e3d06e03d8f7-tmp\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.649894 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.649857    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/516a8da8-1848-4b52-a302-e3d06e03d8f7-tmp\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.650058 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.649926    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bwjq\" (UniqueName: \"kubernetes.io/projected/516a8da8-1848-4b52-a302-e3d06e03d8f7-kube-api-access-4bwjq\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.650058 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.649966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516a8da8-1848-4b52-a302-e3d06e03d8f7-tls-certs\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.652233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.652211    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/516a8da8-1848-4b52-a302-e3d06e03d8f7-tmp\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.652528 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.652499    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/516a8da8-1848-4b52-a302-e3d06e03d8f7-tls-certs\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.656735 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.656706    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bwjq\" (UniqueName: \"kubernetes.io/projected/516a8da8-1848-4b52-a302-e3d06e03d8f7-kube-api-access-4bwjq\") pod \"kube-auth-proxy-5586db6db5-wpksw\" (UID: \"516a8da8-1848-4b52-a302-e3d06e03d8f7\") " pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:35.781893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:35.781814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"
Sep 06 10:34:36.052932 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:36.052820    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5586db6db5-wpksw"]
Sep 06 10:34:36.055784 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:34:36.055760    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516a8da8_1848_4b52_a302_e3d06e03d8f7.slice/crio-b9859ea1f5d5e7ca878bf617f94610034915fd002d526d6bae0ebdd02813aaeb WatchSource:0}: Error finding container b9859ea1f5d5e7ca878bf617f94610034915fd002d526d6bae0ebdd02813aaeb: Status 404 returned error can't find the container with id b9859ea1f5d5e7ca878bf617f94610034915fd002d526d6bae0ebdd02813aaeb
Sep 06 10:34:36.535494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:36.535466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw" event={"ID":"516a8da8-1848-4b52-a302-e3d06e03d8f7","Type":"ContainerStarted","Data":"b9859ea1f5d5e7ca878bf617f94610034915fd002d526d6bae0ebdd02813aaeb"}
Sep 06 10:34:39.550183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:39.550128    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw" event={"ID":"516a8da8-1848-4b52-a302-e3d06e03d8f7","Type":"ContainerStarted","Data":"5f7df5ed525e0294a548772e5ec354213223b6d0080362bba7499e55f13d8aba"}
Sep 06 10:34:39.567377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:39.567330    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5586db6db5-wpksw" podStartSLOduration=1.8615177 podStartE2EDuration="4.567314962s" totalImagesPullingTime="2.705797262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:39.566630807 +0000 UTC m=+262.736480705" watchObservedRunningTime="2026-09-06 10:34:39.567314962 +0000 UTC m=+262.737164834"
Sep 06 10:34:45.532478 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:45.532448    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-59b7bf7857-pzkbz"
Sep 06 10:34:52.022012 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.021960    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"]
Sep 06 10:34:52.032130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.032107    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.037264 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.037242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 06 10:34:52.038351 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.038329    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ppjxq\""
Sep 06 10:34:52.056943 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.056916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"]
Sep 06 10:34:52.093559 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.093524    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.093724 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.093571    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7587s\" (UniqueName: \"kubernetes.io/projected/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-kube-api-access-7587s\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.194502 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.194466    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.194658 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.194513    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7587s\" (UniqueName: \"kubernetes.io/projected/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-kube-api-access-7587s\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.196969 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.196950    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.212296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.212268    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7587s\" (UniqueName: \"kubernetes.io/projected/3ecd2b97-0377-4bc4-a1ca-2c290354a8ee-kube-api-access-7587s\") pod \"llmisvc-controller-manager-777c6bd7d5-d85fq\" (UID: \"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.341669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.341572    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:52.614424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:52.614373    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"]
Sep 06 10:34:52.617127 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:34:52.617099    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3ecd2b97_0377_4bc4_a1ca_2c290354a8ee.slice/crio-54ee673748ec5f889e585b3b9f0cb6873da8f9d7f5f8293ff937479d0bc2f435 WatchSource:0}: Error finding container 54ee673748ec5f889e585b3b9f0cb6873da8f9d7f5f8293ff937479d0bc2f435: Status 404 returned error can't find the container with id 54ee673748ec5f889e585b3b9f0cb6873da8f9d7f5f8293ff937479d0bc2f435
Sep 06 10:34:53.609776 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:53.609738    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq" event={"ID":"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee","Type":"ContainerStarted","Data":"54ee673748ec5f889e585b3b9f0cb6873da8f9d7f5f8293ff937479d0bc2f435"}
Sep 06 10:34:55.620473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:55.620381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq" event={"ID":"3ecd2b97-0377-4bc4-a1ca-2c290354a8ee","Type":"ContainerStarted","Data":"068bd2f2cd087e309bfccd8127b31f8b12075ae51f9a5d8f479bef2ac1c1a34f"}
Sep 06 10:34:55.620473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:55.620468    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:34:55.636640 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:34:55.636590    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq" podStartSLOduration=1.92496036 podStartE2EDuration="4.63657394s" totalImagesPullingTime="2.71161358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:34:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:34:55.635850656 +0000 UTC m=+278.805700530" watchObservedRunningTime="2026-09-06 10:34:55.63657394 +0000 UTC m=+278.806423829"
Sep 06 10:35:17.268322 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:35:17.268296    2573 kubelet.go:1648] "Image garbage collection succeeded"
Sep 06 10:35:26.626649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:35:26.626617    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-d85fq"
Sep 06 10:36:13.810444 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.810397    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"]
Sep 06 10:36:13.813951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.813927    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.815995 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.815965    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 06 10:36:13.816125 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.816051    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 06 10:36:13.816791 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.816773    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t248c\""
Sep 06 10:36:13.816874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.816793    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 06 10:36:13.816874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.816802    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 06 10:36:13.821228 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.821206    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"]
Sep 06 10:36:13.890978 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.890942    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vscj\" (UniqueName: \"kubernetes.io/projected/7939deb1-d286-4728-a093-bedcc4be0510-kube-api-access-6vscj\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.891183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.890995    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7939deb1-d286-4728-a093-bedcc4be0510-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.891183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.891018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.992130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.992092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7939deb1-d286-4728-a093-bedcc4be0510-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.992130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.992138    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.992350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.992210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vscj\" (UniqueName: \"kubernetes.io/projected/7939deb1-d286-4728-a093-bedcc4be0510-kube-api-access-6vscj\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:13.992350 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:36:13.992335    2573 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 06 10:36:13.992420 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:36:13.992405    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert podName:7939deb1-d286-4728-a093-bedcc4be0510 nodeName:}" failed. No retries permitted until 2026-09-06 10:36:14.492384164 +0000 UTC m=+357.662234021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert") pod "kuadrant-console-plugin-598f86468b-69wl2" (UID: "7939deb1-d286-4728-a093-bedcc4be0510") : secret "plugin-serving-cert" not found
Sep 06 10:36:13.992866 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:13.992837    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7939deb1-d286-4728-a093-bedcc4be0510-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:14.001127 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:14.001102    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vscj\" (UniqueName: \"kubernetes.io/projected/7939deb1-d286-4728-a093-bedcc4be0510-kube-api-access-6vscj\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:14.495600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:14.495558    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:14.498026 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:14.498007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/7939deb1-d286-4728-a093-bedcc4be0510-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-69wl2\" (UID: \"7939deb1-d286-4728-a093-bedcc4be0510\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:14.724069 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:14.724036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"
Sep 06 10:36:14.996322 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:14.996295    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2"]
Sep 06 10:36:14.999193 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:36:14.999150    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7939deb1_d286_4728_a093_bedcc4be0510.slice/crio-9f6dfdc34ed385b9cd120fcb689fcc3ff68cf0dcb917621786fca205f7f7c24d WatchSource:0}: Error finding container 9f6dfdc34ed385b9cd120fcb689fcc3ff68cf0dcb917621786fca205f7f7c24d: Status 404 returned error can't find the container with id 9f6dfdc34ed385b9cd120fcb689fcc3ff68cf0dcb917621786fca205f7f7c24d
Sep 06 10:36:15.000843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:15.000828    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:36:15.955639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:15.955578    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2" event={"ID":"7939deb1-d286-4728-a093-bedcc4be0510","Type":"ContainerStarted","Data":"9f6dfdc34ed385b9cd120fcb689fcc3ff68cf0dcb917621786fca205f7f7c24d"}
Sep 06 10:36:19.975066 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:19.975031    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2" event={"ID":"7939deb1-d286-4728-a093-bedcc4be0510","Type":"ContainerStarted","Data":"302d7c2edcec6f579da7109932a2b2a3c14dfec44e35aac8dc273cd726fb4f43"}
Sep 06 10:36:19.990501 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:19.990453    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-69wl2" podStartSLOduration=2.5027480300000002 podStartE2EDuration="6.990405427s" totalImagesPullingTime="4.487657397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:36:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:36:19.989083054 +0000 UTC m=+363.158932924" watchObservedRunningTime="2026-09-06 10:36:19.990405427 +0000 UTC m=+363.160255297"
Sep 06 10:36:24.909852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.909818    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:24.913692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.913665    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:24.915956 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.915932    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-8lm44\""
Sep 06 10:36:24.928302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.928266    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:24.990917 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.990888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:24.990917 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:24.990918    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w6s\" (UniqueName: \"kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.092130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.092096    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.092130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.092132    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w6s\" (UniqueName: \"kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.092511 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.092489    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.100114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.100088    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w6s\" (UniqueName: \"kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s\") pod \"kuadrant-operator-controller-manager-58fc56757c-9x7sm\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") " pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.224985 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.224901    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:25.500838 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.500766    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:25.504800 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:36:25.504765    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c1da84_84b9_484a_9f55_ad3d6d678b08.slice/crio-b105242502de13ae5522e798c59134ee05e56044e2e8d1cd589f8494b6b48351 WatchSource:0}: Error finding container b105242502de13ae5522e798c59134ee05e56044e2e8d1cd589f8494b6b48351: Status 404 returned error can't find the container with id b105242502de13ae5522e798c59134ee05e56044e2e8d1cd589f8494b6b48351
Sep 06 10:36:25.999347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:25.999312    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" event={"ID":"52c1da84-84b9-484a-9f55-ad3d6d678b08","Type":"ContainerStarted","Data":"b105242502de13ae5522e798c59134ee05e56044e2e8d1cd589f8494b6b48351"}
Sep 06 10:36:30.018086 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:30.018047    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" event={"ID":"52c1da84-84b9-484a-9f55-ad3d6d678b08","Type":"ContainerStarted","Data":"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"}
Sep 06 10:36:30.018473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:30.018182    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:30.036839 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:30.036782    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" podStartSLOduration=2.467112797 podStartE2EDuration="6.036761182s" totalImagesPullingTime="3.569648385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:36:30.035761052 +0000 UTC m=+373.205610934" watchObservedRunningTime="2026-09-06 10:36:30.036761182 +0000 UTC m=+373.206611053"
Sep 06 10:36:37.073082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.073041    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:37.073517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.073296    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" podUID="52c1da84-84b9-484a-9f55-ad3d6d678b08" containerName="manager" containerID="cri-o://1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231" gracePeriod=10
Sep 06 10:36:37.075290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.075266    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:37.308150 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.308125    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:37.400909 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.400884    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume\") pod \"52c1da84-84b9-484a-9f55-ad3d6d678b08\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") "
Sep 06 10:36:37.401068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.400993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6w6s\" (UniqueName: \"kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s\") pod \"52c1da84-84b9-484a-9f55-ad3d6d678b08\" (UID: \"52c1da84-84b9-484a-9f55-ad3d6d678b08\") "
Sep 06 10:36:37.401421 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.401392    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "52c1da84-84b9-484a-9f55-ad3d6d678b08" (UID: "52c1da84-84b9-484a-9f55-ad3d6d678b08"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:36:37.403254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.403232    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s" (OuterVolumeSpecName: "kube-api-access-r6w6s") pod "52c1da84-84b9-484a-9f55-ad3d6d678b08" (UID: "52c1da84-84b9-484a-9f55-ad3d6d678b08"). InnerVolumeSpecName "kube-api-access-r6w6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:36:37.502080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.502038    2573 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/52c1da84-84b9-484a-9f55-ad3d6d678b08-extensions-socket-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:36:37.502080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:37.502070    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6w6s\" (UniqueName: \"kubernetes.io/projected/52c1da84-84b9-484a-9f55-ad3d6d678b08-kube-api-access-r6w6s\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:36:38.053397 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.053355    2573 generic.go:356] "Generic (PLEG): container finished" podID="52c1da84-84b9-484a-9f55-ad3d6d678b08" containerID="1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231" exitCode=0
Sep 06 10:36:38.053397 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.053399    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" event={"ID":"52c1da84-84b9-484a-9f55-ad3d6d678b08","Type":"ContainerDied","Data":"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"}
Sep 06 10:36:38.053646 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.053411    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"
Sep 06 10:36:38.053646 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.053436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm" event={"ID":"52c1da84-84b9-484a-9f55-ad3d6d678b08","Type":"ContainerDied","Data":"b105242502de13ae5522e798c59134ee05e56044e2e8d1cd589f8494b6b48351"}
Sep 06 10:36:38.053646 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.053457    2573 scope.go:117] "RemoveContainer" containerID="1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"
Sep 06 10:36:38.062391 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.062372    2573 scope.go:117] "RemoveContainer" containerID="1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"
Sep 06 10:36:38.062712 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:36:38.062694    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231\": container with ID starting with 1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231 not found: ID does not exist" containerID="1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"
Sep 06 10:36:38.062773 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.062721    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231"} err="failed to get container status \"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231\": rpc error: code = NotFound desc = could not find container \"1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231\": container with ID starting with 1a9fc768632b0cb13125a213d5d3516f8d4192355461f6200abb6dabc5d50231 not found: ID does not exist"
Sep 06 10:36:38.077207 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.077156    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:38.083591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:38.083565    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-58fc56757c-9x7sm"]
Sep 06 10:36:39.352469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:36:39.352438    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52c1da84-84b9-484a-9f55-ad3d6d678b08" path="/var/lib/kubelet/pods/52c1da84-84b9-484a-9f55-ad3d6d678b08/volumes"
Sep 06 10:37:06.925269 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.925233    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:06.925715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.925553    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52c1da84-84b9-484a-9f55-ad3d6d678b08" containerName="manager"
Sep 06 10:37:06.925715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.925564    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="52c1da84-84b9-484a-9f55-ad3d6d678b08" containerName="manager"
Sep 06 10:37:06.925715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.925617    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="52c1da84-84b9-484a-9f55-ad3d6d678b08" containerName="manager"
Sep 06 10:37:06.950981 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.950957    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:06.951130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.951059    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:06.952837 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.952808    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr8wv\" (UniqueName: \"kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:06.952963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.952880    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:06.953328 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:06.953304    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 06 10:37:07.017985 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.017955    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:07.053732 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.053703    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:07.053888 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.053782    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr8wv\" (UniqueName: \"kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:07.054386 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.054370    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:07.065029 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.064997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr8wv\" (UniqueName: \"kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv\") pod \"limitador-limitador-859c4d96d6-9g6sk\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:07.261116 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.261036    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:07.530086 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.530015    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:07.533976 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:37:07.533950    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa21cf7_c6a2_46d1_9b87_c7f1fdc1a4f6.slice/crio-d38c970a2b243cde797d0fcc940b3e683bb5f0e12826e43af159a0b4b419d452 WatchSource:0}: Error finding container d38c970a2b243cde797d0fcc940b3e683bb5f0e12826e43af159a0b4b419d452: Status 404 returned error can't find the container with id d38c970a2b243cde797d0fcc940b3e683bb5f0e12826e43af159a0b4b419d452
Sep 06 10:37:07.723734 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.723700    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:07.728299 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.728279    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:07.730429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.730409    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mjjjs\""
Sep 06 10:37:07.733279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.733257    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:07.758471 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.758435    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d24m8\" (UniqueName: \"kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8\") pod \"authorino-b54df94b4-pwkzp\" (UID: \"854c63f5-70f1-4900-99c9-dba520a0d94b\") " pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:07.859544 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.859454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d24m8\" (UniqueName: \"kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8\") pod \"authorino-b54df94b4-pwkzp\" (UID: \"854c63f5-70f1-4900-99c9-dba520a0d94b\") " pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:07.868899 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:07.868866    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d24m8\" (UniqueName: \"kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8\") pod \"authorino-b54df94b4-pwkzp\" (UID: \"854c63f5-70f1-4900-99c9-dba520a0d94b\") " pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:08.038368 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:08.038332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:08.179650 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:08.179613    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" event={"ID":"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6","Type":"ContainerStarted","Data":"d38c970a2b243cde797d0fcc940b3e683bb5f0e12826e43af159a0b4b419d452"}
Sep 06 10:37:08.311262 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:37:08.311231    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod854c63f5_70f1_4900_99c9_dba520a0d94b.slice/crio-610f1accae48a8a323f23c09ebe4d3cfa4fe3cc9001c29704d148895a6451db7 WatchSource:0}: Error finding container 610f1accae48a8a323f23c09ebe4d3cfa4fe3cc9001c29704d148895a6451db7: Status 404 returned error can't find the container with id 610f1accae48a8a323f23c09ebe4d3cfa4fe3cc9001c29704d148895a6451db7
Sep 06 10:37:08.312202 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:08.312178    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:09.186764 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:09.186721    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-pwkzp" event={"ID":"854c63f5-70f1-4900-99c9-dba520a0d94b","Type":"ContainerStarted","Data":"610f1accae48a8a323f23c09ebe4d3cfa4fe3cc9001c29704d148895a6451db7"}
Sep 06 10:37:11.197342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:11.197304    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-pwkzp" event={"ID":"854c63f5-70f1-4900-99c9-dba520a0d94b","Type":"ContainerStarted","Data":"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"}
Sep 06 10:37:11.211833 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:11.211786    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-pwkzp" podStartSLOduration=2.3313067690000002 podStartE2EDuration="4.211768912s" totalImagesPullingTime="1.880462143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:11.210435793 +0000 UTC m=+414.380285663" watchObservedRunningTime="2026-09-06 10:37:11.211768912 +0000 UTC m=+414.381618783"
Sep 06 10:37:11.243566 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:11.243532    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:13.209062 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.209021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" event={"ID":"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6","Type":"ContainerStarted","Data":"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"}
Sep 06 10:37:13.209542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.209105    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:13.209542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.209220    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-pwkzp" podUID="854c63f5-70f1-4900-99c9-dba520a0d94b" containerName="authorino" containerID="cri-o://8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923" gracePeriod=30
Sep 06 10:37:13.224348 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.224307    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" podStartSLOduration=1.719313057 podStartE2EDuration="7.224291396s" totalImagesPullingTime="5.504978339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:13.223108232 +0000 UTC m=+416.392958102" watchObservedRunningTime="2026-09-06 10:37:13.224291396 +0000 UTC m=+416.394141246"
Sep 06 10:37:13.446810 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.446785    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:13.502255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.502215    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d24m8\" (UniqueName: \"kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8\") pod \"854c63f5-70f1-4900-99c9-dba520a0d94b\" (UID: \"854c63f5-70f1-4900-99c9-dba520a0d94b\") "
Sep 06 10:37:13.504438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.504408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8" (OuterVolumeSpecName: "kube-api-access-d24m8") pod "854c63f5-70f1-4900-99c9-dba520a0d94b" (UID: "854c63f5-70f1-4900-99c9-dba520a0d94b"). InnerVolumeSpecName "kube-api-access-d24m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:13.603357 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:13.603263    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d24m8\" (UniqueName: \"kubernetes.io/projected/854c63f5-70f1-4900-99c9-dba520a0d94b-kube-api-access-d24m8\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:14.214253 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.214213    2573 generic.go:356] "Generic (PLEG): container finished" podID="854c63f5-70f1-4900-99c9-dba520a0d94b" containerID="8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923" exitCode=0
Sep 06 10:37:14.214659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.214269    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-pwkzp"
Sep 06 10:37:14.214659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.214290    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-pwkzp" event={"ID":"854c63f5-70f1-4900-99c9-dba520a0d94b","Type":"ContainerDied","Data":"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"}
Sep 06 10:37:14.214659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.214328    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-pwkzp" event={"ID":"854c63f5-70f1-4900-99c9-dba520a0d94b","Type":"ContainerDied","Data":"610f1accae48a8a323f23c09ebe4d3cfa4fe3cc9001c29704d148895a6451db7"}
Sep 06 10:37:14.214659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.214344    2573 scope.go:117] "RemoveContainer" containerID="8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"
Sep 06 10:37:14.222479 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.222463    2573 scope.go:117] "RemoveContainer" containerID="8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"
Sep 06 10:37:14.222750 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:14.222731    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923\": container with ID starting with 8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923 not found: ID does not exist" containerID="8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"
Sep 06 10:37:14.222815 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.222757    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923"} err="failed to get container status \"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923\": rpc error: code = NotFound desc = could not find container \"8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923\": container with ID starting with 8ce3e610c0ddc03136d3eb9b14f5ec571b776f1cb3092c8e313485235a2dd923 not found: ID does not exist"
Sep 06 10:37:14.232706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.232686    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:14.234569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:14.234549    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-pwkzp"]
Sep 06 10:37:15.352429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:15.352396    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="854c63f5-70f1-4900-99c9-dba520a0d94b" path="/var/lib/kubelet/pods/854c63f5-70f1-4900-99c9-dba520a0d94b/volumes"
Sep 06 10:37:20.237729 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.237691    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:20.238257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.237859    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" podUID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" containerName="limitador" containerID="cri-o://9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1" gracePeriod=30
Sep 06 10:37:20.240156 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.240130    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:20.781015 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.780989    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:20.864661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.864575    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file\") pod \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") "
Sep 06 10:37:20.864661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.864623    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr8wv\" (UniqueName: \"kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv\") pod \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\" (UID: \"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6\") "
Sep 06 10:37:20.864909 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.864888    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file" (OuterVolumeSpecName: "config-file") pod "eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" (UID: "eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:37:20.866793 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.866775    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv" (OuterVolumeSpecName: "kube-api-access-vr8wv") pod "eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" (UID: "eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6"). InnerVolumeSpecName "kube-api-access-vr8wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:20.966050 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.965998    2573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-config-file\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:20.966050 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:20.966043    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr8wv\" (UniqueName: \"kubernetes.io/projected/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6-kube-api-access-vr8wv\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:21.245642 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.245602    2573 generic.go:356] "Generic (PLEG): container finished" podID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" containerID="9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1" exitCode=0
Sep 06 10:37:21.246082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.245652    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"
Sep 06 10:37:21.246082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.245682    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" event={"ID":"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6","Type":"ContainerDied","Data":"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"}
Sep 06 10:37:21.246082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.245734    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-9g6sk" event={"ID":"eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6","Type":"ContainerDied","Data":"d38c970a2b243cde797d0fcc940b3e683bb5f0e12826e43af159a0b4b419d452"}
Sep 06 10:37:21.246082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.245771    2573 scope.go:117] "RemoveContainer" containerID="9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"
Sep 06 10:37:21.257116 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.257094    2573 scope.go:117] "RemoveContainer" containerID="9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"
Sep 06 10:37:21.257450 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:21.257424    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1\": container with ID starting with 9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1 not found: ID does not exist" containerID="9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"
Sep 06 10:37:21.257542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.257456    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1"} err="failed to get container status \"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1\": rpc error: code = NotFound desc = could not find container \"9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1\": container with ID starting with 9bae8d921da61745be7d450087e479c5a75ed557c444faf516a982bf92e6c8a1 not found: ID does not exist"
Sep 06 10:37:21.270987 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.270962    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:21.273068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.273044    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-9g6sk"]
Sep 06 10:37:21.352114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:21.352082    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" path="/var/lib/kubelet/pods/eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6/volumes"
Sep 06 10:37:24.910822 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.910784    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"]
Sep 06 10:37:24.911312 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911274    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" containerName="limitador"
Sep 06 10:37:24.911312 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911292    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" containerName="limitador"
Sep 06 10:37:24.911312 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911309    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="854c63f5-70f1-4900-99c9-dba520a0d94b" containerName="authorino"
Sep 06 10:37:24.911457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911317    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="854c63f5-70f1-4900-99c9-dba520a0d94b" containerName="authorino"
Sep 06 10:37:24.911457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911414    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaa21cf7-c6a2-46d1-9b87-c7f1fdc1a4f6" containerName="limitador"
Sep 06 10:37:24.911457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.911428    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="854c63f5-70f1-4900-99c9-dba520a0d94b" containerName="authorino"
Sep 06 10:37:24.915989 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.915967    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:24.918114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.918093    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 06 10:37:24.918114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.918106    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 06 10:37:24.918410 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.918395    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 06 10:37:24.918831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.918817    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-l78qr\""
Sep 06 10:37:24.925104 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:24.925081    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"]
Sep 06 10:37:25.005981 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.005941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a67ed0b9-27a8-4dba-9697-56dc26b3308f-data\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.005981 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.005986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55tws\" (UniqueName: \"kubernetes.io/projected/a67ed0b9-27a8-4dba-9697-56dc26b3308f-kube-api-access-55tws\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.106564 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.106533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a67ed0b9-27a8-4dba-9697-56dc26b3308f-data\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.106712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.106569    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55tws\" (UniqueName: \"kubernetes.io/projected/a67ed0b9-27a8-4dba-9697-56dc26b3308f-kube-api-access-55tws\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.106967 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.106948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a67ed0b9-27a8-4dba-9697-56dc26b3308f-data\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.114590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.114565    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55tws\" (UniqueName: \"kubernetes.io/projected/a67ed0b9-27a8-4dba-9697-56dc26b3308f-kube-api-access-55tws\") pod \"postgres-69fb4594cf-mr7vk\" (UID: \"a67ed0b9-27a8-4dba-9697-56dc26b3308f\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.227667 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.227584    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:25.500315 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:25.500237    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"]
Sep 06 10:37:25.503097 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:37:25.503072    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda67ed0b9_27a8_4dba_9697_56dc26b3308f.slice/crio-cd03b36b85c170be2d9637e18add42a08f4a5e38f939b45b6fcd885e93f5b532 WatchSource:0}: Error finding container cd03b36b85c170be2d9637e18add42a08f4a5e38f939b45b6fcd885e93f5b532: Status 404 returned error can't find the container with id cd03b36b85c170be2d9637e18add42a08f4a5e38f939b45b6fcd885e93f5b532
Sep 06 10:37:26.273101 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:26.273025    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk" event={"ID":"a67ed0b9-27a8-4dba-9697-56dc26b3308f","Type":"ContainerStarted","Data":"cd03b36b85c170be2d9637e18add42a08f4a5e38f939b45b6fcd885e93f5b532"}
Sep 06 10:37:30.294066 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:30.294024    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk" event={"ID":"a67ed0b9-27a8-4dba-9697-56dc26b3308f","Type":"ContainerStarted","Data":"0741b1211cfc72d83e75f40ef29e5b097839ef76e902ef1736b11e921b1b0629"}
Sep 06 10:37:30.294493 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:30.294152    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:30.312571 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:30.312504    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk" podStartSLOduration=1.731367114 podStartE2EDuration="6.312483761s" totalImagesPullingTime="4.581116647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:30.311923163 +0000 UTC m=+433.481773033" watchObservedRunningTime="2026-09-06 10:37:30.312483761 +0000 UTC m=+433.482333636"
Sep 06 10:37:36.327019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.326991    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-mr7vk"
Sep 06 10:37:36.849960 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.849928    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-k55g4"]
Sep 06 10:37:36.856055 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.856031    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:36.858451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.858429    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mjjjs\""
Sep 06 10:37:36.860289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.860267    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-k55g4"]
Sep 06 10:37:36.992707 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:36.992672    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-k55g4"]
Sep 06 10:37:36.992920 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:36.992903    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6rtwq], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5dc7ccb5f-k55g4" podUID="58d7927c-af3b-4f2a-a041-570e5c1c1722"
Sep 06 10:37:37.017459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.017422    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-687bd45585-jp9rd"]
Sep 06 10:37:37.017745 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.017721    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rtwq\" (UniqueName: \"kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq\") pod \"authorino-5dc7ccb5f-k55g4\" (UID: \"58d7927c-af3b-4f2a-a041-570e5c1c1722\") " pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:37.021597 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.021582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.023893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.023875    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 10:37:37.030182 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.030126    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-687bd45585-jp9rd"]
Sep 06 10:37:37.088294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.088256    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-jp9rd"]
Sep 06 10:37:37.088518 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:37.088500    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-r9glk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-687bd45585-jp9rd" podUID="21501458-6962-456a-a03f-35f627e0ea1f"
Sep 06 10:37:37.118994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.118914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.118994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.118946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9glk\" (UniqueName: \"kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.119207 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.119033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rtwq\" (UniqueName: \"kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq\") pod \"authorino-5dc7ccb5f-k55g4\" (UID: \"58d7927c-af3b-4f2a-a041-570e5c1c1722\") " pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:37.130287 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.130263    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rtwq\" (UniqueName: \"kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq\") pod \"authorino-5dc7ccb5f-k55g4\" (UID: \"58d7927c-af3b-4f2a-a041-570e5c1c1722\") " pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:37.219501 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.219473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.219501 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.219503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9glk\" (UniqueName: \"kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.221843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.221812    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.228732 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.228710    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9glk\" (UniqueName: \"kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk\") pod \"authorino-687bd45585-jp9rd\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") " pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.322652 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.322623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:37.322805 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.322623    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.330462 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.330445    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:37.333133 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.333120    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:37.421390 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.421367    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rtwq\" (UniqueName: \"kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq\") pod \"58d7927c-af3b-4f2a-a041-570e5c1c1722\" (UID: \"58d7927c-af3b-4f2a-a041-570e5c1c1722\") "
Sep 06 10:37:37.421553 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.421420    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert\") pod \"21501458-6962-456a-a03f-35f627e0ea1f\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") "
Sep 06 10:37:37.421553 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.421457    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9glk\" (UniqueName: \"kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk\") pod \"21501458-6962-456a-a03f-35f627e0ea1f\" (UID: \"21501458-6962-456a-a03f-35f627e0ea1f\") "
Sep 06 10:37:37.423625 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.423599    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq" (OuterVolumeSpecName: "kube-api-access-6rtwq") pod "58d7927c-af3b-4f2a-a041-570e5c1c1722" (UID: "58d7927c-af3b-4f2a-a041-570e5c1c1722"). InnerVolumeSpecName "kube-api-access-6rtwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:37.423625 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.423605    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "21501458-6962-456a-a03f-35f627e0ea1f" (UID: "21501458-6962-456a-a03f-35f627e0ea1f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:37.423781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.423700    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk" (OuterVolumeSpecName: "kube-api-access-r9glk") pod "21501458-6962-456a-a03f-35f627e0ea1f" (UID: "21501458-6962-456a-a03f-35f627e0ea1f"). InnerVolumeSpecName "kube-api-access-r9glk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:37.522864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.522829    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rtwq\" (UniqueName: \"kubernetes.io/projected/58d7927c-af3b-4f2a-a041-570e5c1c1722-kube-api-access-6rtwq\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:37.522864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.522858    2573 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/21501458-6962-456a-a03f-35f627e0ea1f-tls-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:37.522864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:37.522868    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9glk\" (UniqueName: \"kubernetes.io/projected/21501458-6962-456a-a03f-35f627e0ea1f-kube-api-access-r9glk\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:38.326420 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.326347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-k55g4"
Sep 06 10:37:38.326673 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.326347    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-687bd45585-jp9rd"
Sep 06 10:37:38.358278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.358250    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-687bd45585-jp9rd"]
Sep 06 10:37:38.360668 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.360645    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-687bd45585-jp9rd"]
Sep 06 10:37:38.382838 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.382802    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-k55g4"]
Sep 06 10:37:38.392153 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:38.392131    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-k55g4"]
Sep 06 10:37:39.352080 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:39.352052    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21501458-6962-456a-a03f-35f627e0ea1f" path="/var/lib/kubelet/pods/21501458-6962-456a-a03f-35f627e0ea1f/volumes"
Sep 06 10:37:39.352350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:39.352332    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d7927c-af3b-4f2a-a041-570e5c1c1722" path="/var/lib/kubelet/pods/58d7927c-af3b-4f2a-a041-570e5c1c1722/volumes"
Sep 06 10:37:44.052465 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.052432    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:37:44.059944 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.059925    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.064410 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.064392    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:37:44.064669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.064646    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:37:44.064845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.064826    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h29zj\""
Sep 06 10:37:44.065223 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.065207    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:37:44.072654 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.072633    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:37:44.180343 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.180305    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nk5x\" (UniqueName: \"kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.180539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.180374    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.180539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.180428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.220859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.220829    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:37:44.224117 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.224099    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.232659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.232634    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:37:44.281742 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.281713    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.281903 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.281765    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nk5x\" (UniqueName: \"kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.281903 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.281808    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.281903 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:44.281866    2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 06 10:37:44.281903 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:44.281898    2573 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 06 10:37:44.282035 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:44.281933    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs podName:0f3baf01-81c7-4bce-ae14-723a9c86e582 nodeName:}" failed. No retries permitted until 2026-09-06 10:37:44.781916216 +0000 UTC m=+447.951766065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs") pod "maas-controller-d678f5c67-b4n2g" (UID: "0f3baf01-81c7-4bce-ae14-723a9c86e582") : secret "maas-controller-metrics-tls" not found
Sep 06 10:37:44.282035 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:37:44.281954    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert podName:0f3baf01-81c7-4bce-ae14-723a9c86e582 nodeName:}" failed. No retries permitted until 2026-09-06 10:37:44.78194093 +0000 UTC m=+447.951790780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert") pod "maas-controller-d678f5c67-b4n2g" (UID: "0f3baf01-81c7-4bce-ae14-723a9c86e582") : secret "maas-controller-webhook-cert" not found
Sep 06 10:37:44.295235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.295204    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nk5x\" (UniqueName: \"kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.383102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.383016    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.383102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.383051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsc9\" (UniqueName: \"kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.383102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.383088    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.483621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.483582    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.483784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.483697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.483784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.483730    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsc9\" (UniqueName: \"kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.486186 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.486146    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.486285 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.486216    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.491875 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.491855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsc9\" (UniqueName: \"kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9\") pod \"maas-controller-5fb95fb9bd-qmzcd\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") " pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.534906 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.534882    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:44.785835 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.785806    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.785998 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.785844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.788306 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.788279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.788412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.788333    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") pod \"maas-controller-d678f5c67-b4n2g\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") " pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:44.803341 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.803316    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:37:44.806636 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:37:44.806617    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa7cbc33_4eb8_4a54_9ad8_ceb40e791b5e.slice/crio-99a1f573f8e0d218548b67064e008368465cea7f2bf2ba1bd7cd9fbd163fb22f WatchSource:0}: Error finding container 99a1f573f8e0d218548b67064e008368465cea7f2bf2ba1bd7cd9fbd163fb22f: Status 404 returned error can't find the container with id 99a1f573f8e0d218548b67064e008368465cea7f2bf2ba1bd7cd9fbd163fb22f
Sep 06 10:37:44.969528 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:44.969492    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:45.239784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:45.239754    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:37:45.242688 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:37:45.242654    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3baf01_81c7_4bce_ae14_723a9c86e582.slice/crio-dd3c6448cea1b4b76bdd52024cfee8caefbecd57f5774ed944bde89d4b431ec8 WatchSource:0}: Error finding container dd3c6448cea1b4b76bdd52024cfee8caefbecd57f5774ed944bde89d4b431ec8: Status 404 returned error can't find the container with id dd3c6448cea1b4b76bdd52024cfee8caefbecd57f5774ed944bde89d4b431ec8
Sep 06 10:37:45.368887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:45.368847    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" event={"ID":"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e","Type":"ContainerStarted","Data":"99a1f573f8e0d218548b67064e008368465cea7f2bf2ba1bd7cd9fbd163fb22f"}
Sep 06 10:37:45.370496 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:45.370466    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d678f5c67-b4n2g" event={"ID":"0f3baf01-81c7-4bce-ae14-723a9c86e582","Type":"ContainerStarted","Data":"dd3c6448cea1b4b76bdd52024cfee8caefbecd57f5774ed944bde89d4b431ec8"}
Sep 06 10:37:48.391052 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.391015    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" event={"ID":"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e","Type":"ContainerStarted","Data":"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"}
Sep 06 10:37:48.391499 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.391121    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:48.392554 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.392531    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d678f5c67-b4n2g" event={"ID":"0f3baf01-81c7-4bce-ae14-723a9c86e582","Type":"ContainerStarted","Data":"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"}
Sep 06 10:37:48.392672 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.392655    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:48.413283 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.413236    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" podStartSLOduration=1.9017095720000001 podStartE2EDuration="4.413222162s" totalImagesPullingTime="2.51151259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:48.411711346 +0000 UTC m=+451.581561218" watchObservedRunningTime="2026-09-06 10:37:48.413222162 +0000 UTC m=+451.583072032"
Sep 06 10:37:48.428854 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:48.428808    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d678f5c67-b4n2g" podStartSLOduration=2.346345889 podStartE2EDuration="4.428793144s" totalImagesPullingTime="2.082447255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:37:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:37:48.427538864 +0000 UTC m=+451.597388736" watchObservedRunningTime="2026-09-06 10:37:48.428793144 +0000 UTC m=+451.598643014"
Sep 06 10:37:59.399254 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.399225    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:37:59.399613 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.399291    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:59.465515 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.465482    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:37:59.465724 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.465679    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d678f5c67-b4n2g" podUID="0f3baf01-81c7-4bce-ae14-723a9c86e582" containerName="manager" containerID="cri-o://0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6" gracePeriod=30
Sep 06 10:37:59.706882 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.706858    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:37:59.804324 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.804289    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") pod \"0f3baf01-81c7-4bce-ae14-723a9c86e582\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") "
Sep 06 10:37:59.804515 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.804342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") pod \"0f3baf01-81c7-4bce-ae14-723a9c86e582\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") "
Sep 06 10:37:59.804515 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.804395    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nk5x\" (UniqueName: \"kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x\") pod \"0f3baf01-81c7-4bce-ae14-723a9c86e582\" (UID: \"0f3baf01-81c7-4bce-ae14-723a9c86e582\") "
Sep 06 10:37:59.806582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.806546    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0f3baf01-81c7-4bce-ae14-723a9c86e582" (UID: "0f3baf01-81c7-4bce-ae14-723a9c86e582"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:59.806684 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.806594    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x" (OuterVolumeSpecName: "kube-api-access-7nk5x") pod "0f3baf01-81c7-4bce-ae14-723a9c86e582" (UID: "0f3baf01-81c7-4bce-ae14-723a9c86e582"). InnerVolumeSpecName "kube-api-access-7nk5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:37:59.806684 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.806624    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0f3baf01-81c7-4bce-ae14-723a9c86e582" (UID: "0f3baf01-81c7-4bce-ae14-723a9c86e582"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:37:59.905330 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.905293    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-webhook-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.905330 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.905321    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f3baf01-81c7-4bce-ae14-723a9c86e582-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:37:59.905330 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:37:59.905330    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nk5x\" (UniqueName: \"kubernetes.io/projected/0f3baf01-81c7-4bce-ae14-723a9c86e582-kube-api-access-7nk5x\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:38:00.445339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.445305    2573 generic.go:356] "Generic (PLEG): container finished" podID="0f3baf01-81c7-4bce-ae14-723a9c86e582" containerID="0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6" exitCode=0
Sep 06 10:38:00.445751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.445346    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d678f5c67-b4n2g"
Sep 06 10:38:00.445751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.445390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d678f5c67-b4n2g" event={"ID":"0f3baf01-81c7-4bce-ae14-723a9c86e582","Type":"ContainerDied","Data":"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"}
Sep 06 10:38:00.445751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.445436    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d678f5c67-b4n2g" event={"ID":"0f3baf01-81c7-4bce-ae14-723a9c86e582","Type":"ContainerDied","Data":"dd3c6448cea1b4b76bdd52024cfee8caefbecd57f5774ed944bde89d4b431ec8"}
Sep 06 10:38:00.445751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.445457    2573 scope.go:117] "RemoveContainer" containerID="0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"
Sep 06 10:38:00.453699 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.453685    2573 scope.go:117] "RemoveContainer" containerID="0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"
Sep 06 10:38:00.453973 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:38:00.453957    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6\": container with ID starting with 0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6 not found: ID does not exist" containerID="0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"
Sep 06 10:38:00.454018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.453982    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6"} err="failed to get container status \"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6\": rpc error: code = NotFound desc = could not find container \"0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6\": container with ID starting with 0d8ce067e53e79e0a5f7b497c26cc3543b87337dfe29cea458057b9c8d4065a6 not found: ID does not exist"
Sep 06 10:38:00.465567 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.465545    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:38:00.468871 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:00.468851    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d678f5c67-b4n2g"]
Sep 06 10:38:01.352426 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:01.352398    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f3baf01-81c7-4bce-ae14-723a9c86e582" path="/var/lib/kubelet/pods/0f3baf01-81c7-4bce-ae14-723a9c86e582/volumes"
Sep 06 10:38:21.783584 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.783543    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 10:38:21.784077 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.784031    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3baf01-81c7-4bce-ae14-723a9c86e582" containerName="manager"
Sep 06 10:38:21.784077 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.784051    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3baf01-81c7-4bce-ae14-723a9c86e582" containerName="manager"
Sep 06 10:38:21.784234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.784137    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3baf01-81c7-4bce-ae14-723a9c86e582" containerName="manager"
Sep 06 10:38:21.788958 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.788936    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.791711 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.791692    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-q972h\""
Sep 06 10:38:21.791711 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.791702    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 06 10:38:21.791874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.791694    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 06 10:38:21.797024 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.796997    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 10:38:21.897931 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.897898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.898085 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.897943    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgscd\" (UniqueName: \"kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.898085 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.898013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.898225 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.898089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.999093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.999060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.999287 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.999109    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.999287 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.999131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgscd\" (UniqueName: \"kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.999287 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.999188    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:21.999840 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:21.999816    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:22.001780 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.001754    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:22.001889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.001845    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:22.006657 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.006635    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgscd\" (UniqueName: \"kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd\") pod \"maas-api-bd6c55749-b4hvr\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") " pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:22.099805 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.099732    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:22.245562 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.245529    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"]
Sep 06 10:38:22.250041 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.250024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.252217 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.252196    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 06 10:38:22.252351 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.252273    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n8tp6\""
Sep 06 10:38:22.258038 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.258017    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"]
Sep 06 10:38:22.370157 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.370131    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 10:38:22.371988 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:38:22.371956    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffad3759_5647_49bb_a1aa_9d82d9380be6.slice/crio-3564cb2955f5d31dae2792ec82088590b217eb1169e22ebcf2c0030c53cc812a WatchSource:0}: Error finding container 3564cb2955f5d31dae2792ec82088590b217eb1169e22ebcf2c0030c53cc812a: Status 404 returned error can't find the container with id 3564cb2955f5d31dae2792ec82088590b217eb1169e22ebcf2c0030c53cc812a
Sep 06 10:38:22.401944 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.401912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.402054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.401974    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l56kx\" (UniqueName: \"kubernetes.io/projected/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-kube-api-access-l56kx\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.502916 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.502883    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.503115 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.502938    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l56kx\" (UniqueName: \"kubernetes.io/projected/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-kube-api-access-l56kx\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.503664 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.503638    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.513732 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.513707    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l56kx\" (UniqueName: \"kubernetes.io/projected/3a9c6cc5-aa86-4c1e-9417-f68aaf204a71-kube-api-access-l56kx\") pod \"payload-pre-processing-59fbc47bf8-b4xt6\" (UID: \"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.541016 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.540985    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" event={"ID":"ffad3759-5647-49bb-a1aa-9d82d9380be6","Type":"ContainerStarted","Data":"3564cb2955f5d31dae2792ec82088590b217eb1169e22ebcf2c0030c53cc812a"}
Sep 06 10:38:22.562395 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.562370    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:22.834488 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:22.834455    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"]
Sep 06 10:38:22.837422 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:38:22.837396    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9c6cc5_aa86_4c1e_9417_f68aaf204a71.slice/crio-0cd083bac06f7b5a32f8d58e2a259b36f6e448cfd1e7468a7c0a62855dd3262d WatchSource:0}: Error finding container 0cd083bac06f7b5a32f8d58e2a259b36f6e448cfd1e7468a7c0a62855dd3262d: Status 404 returned error can't find the container with id 0cd083bac06f7b5a32f8d58e2a259b36f6e448cfd1e7468a7c0a62855dd3262d
Sep 06 10:38:23.549296 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:23.549258    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6" event={"ID":"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71","Type":"ContainerStarted","Data":"0cd083bac06f7b5a32f8d58e2a259b36f6e448cfd1e7468a7c0a62855dd3262d"}
Sep 06 10:38:24.554604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.554569    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" event={"ID":"ffad3759-5647-49bb-a1aa-9d82d9380be6","Type":"ContainerStarted","Data":"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"}
Sep 06 10:38:24.555024 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.554733    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:24.556100 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.556079    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6" event={"ID":"3a9c6cc5-aa86-4c1e-9417-f68aaf204a71","Type":"ContainerStarted","Data":"e36165501497b83f7d61e7e1f806291c37dba625a2d5f3dc3999bfadf8eb7232"}
Sep 06 10:38:24.556218 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.556149    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:24.571097 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.571055    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" podStartSLOduration=1.9907138720000002 podStartE2EDuration="3.571040851s" totalImagesPullingTime="1.580326979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:24.569093054 +0000 UTC m=+487.738942926" watchObservedRunningTime="2026-09-06 10:38:24.571040851 +0000 UTC m=+487.740890720"
Sep 06 10:38:24.584703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:24.584662    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6" podStartSLOduration=1.01699924 podStartE2EDuration="2.58464874s" totalImagesPullingTime="1.5676495s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:24.583790671 +0000 UTC m=+487.753640542" watchObservedRunningTime="2026-09-06 10:38:24.58464874 +0000 UTC m=+487.754498610"
Sep 06 10:38:30.565336 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:30.565306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 10:38:35.562338 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:35.562310    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-b4xt6"
Sep 06 10:38:47.618786 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.618746    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"]
Sep 06 10:38:47.622113 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.622093    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.624292 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.624263    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 06 10:38:47.624417 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.624349    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 06 10:38:47.625255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.625235    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h68bt\""
Sep 06 10:38:47.625255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.625240    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 06 10:38:47.632154 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.632136    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"]
Sep 06 10:38:47.724945 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.724914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae440408-3afe-4a78-9822-15145e540b8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.724945 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.724947    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.725144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.724986    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.725144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.725056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.725144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.725096    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.725144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.725132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z5f4\" (UniqueName: \"kubernetes.io/projected/ae440408-3afe-4a78-9822-15145e540b8e-kube-api-access-4z5f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826506 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826473    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826656 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z5f4\" (UniqueName: \"kubernetes.io/projected/ae440408-3afe-4a78-9822-15145e540b8e-kube-api-access-4z5f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826656 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826572    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae440408-3afe-4a78-9822-15145e540b8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826656 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826656 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826835 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826688    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.826945 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826910    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.827083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.826978    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.827083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.827027    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.828849 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.828830    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae440408-3afe-4a78-9822-15145e540b8e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.829095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.829077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae440408-3afe-4a78-9822-15145e540b8e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.833617 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.833596    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z5f4\" (UniqueName: \"kubernetes.io/projected/ae440408-3afe-4a78-9822-15145e540b8e-kube-api-access-4z5f4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-dtqzv\" (UID: \"ae440408-3afe-4a78-9822-15145e540b8e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:47.932718 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:47.932677    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:48.201392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:48.201356    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"]
Sep 06 10:38:48.203598 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:38:48.203570    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae440408_3afe_4a78_9822_15145e540b8e.slice/crio-3ca6dae00203bfe24fc284c5541ae0972e376fd41c4c9f35010bad8357542475 WatchSource:0}: Error finding container 3ca6dae00203bfe24fc284c5541ae0972e376fd41c4c9f35010bad8357542475: Status 404 returned error can't find the container with id 3ca6dae00203bfe24fc284c5541ae0972e376fd41c4c9f35010bad8357542475
Sep 06 10:38:48.661905 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:48.661862    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv" event={"ID":"ae440408-3afe-4a78-9822-15145e540b8e","Type":"ContainerStarted","Data":"3ca6dae00203bfe24fc284c5541ae0972e376fd41c4c9f35010bad8357542475"}
Sep 06 10:38:49.667783 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:49.667747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv" event={"ID":"ae440408-3afe-4a78-9822-15145e540b8e","Type":"ContainerStarted","Data":"08fbe8f02100fb2f2ef9099bb8a80dfaf834eaaa16444018268990e8789faf91"}
Sep 06 10:38:49.685992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:49.685943    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv" podStartSLOduration=1.68925422 podStartE2EDuration="2.685927908s" totalImagesPullingTime="996.673688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:38:49.684405074 +0000 UTC m=+512.854254940" watchObservedRunningTime="2026-09-06 10:38:49.685927908 +0000 UTC m=+512.855777778"
Sep 06 10:38:57.933694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:57.933656    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:57.934082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:57.933739    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:57.945867 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:57.945840    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:38:58.717349 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:38:58.717321    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-dtqzv"
Sep 06 10:39:21.724567 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.724538    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"]
Sep 06 10:39:21.728083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.728064    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.730316 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.730294    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 06 10:39:21.736559 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.736528    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"]
Sep 06 10:39:21.827708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827679    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7j5s\" (UniqueName: \"kubernetes.io/projected/19158881-0979-4ec5-97d4-0f433497dca5-kube-api-access-f7j5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.827869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827716    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.827869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.827869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.827869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827804    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19158881-0979-4ec5-97d4-0f433497dca5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.827869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.827825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929043 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929043 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929045    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929060    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19158881-0979-4ec5-97d4-0f433497dca5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929214    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7j5s\" (UniqueName: \"kubernetes.io/projected/19158881-0979-4ec5-97d4-0f433497dca5-kube-api-access-f7j5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929516 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929542    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.929661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.929641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.931600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.931574    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19158881-0979-4ec5-97d4-0f433497dca5-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.931771 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.931757    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19158881-0979-4ec5-97d4-0f433497dca5-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:21.936552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:21.936527    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7j5s\" (UniqueName: \"kubernetes.io/projected/19158881-0979-4ec5-97d4-0f433497dca5-kube-api-access-f7j5s\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7\" (UID: \"19158881-0979-4ec5-97d4-0f433497dca5\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:22.038709 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:22.038618    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:22.309106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:22.308930    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"]
Sep 06 10:39:22.311880 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:39:22.311849    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19158881_0979_4ec5_97d4_0f433497dca5.slice/crio-6b75993c5d464f2b8dcc9f9116b27b26a70e70ecc92f816bac0d3f3fbba932bc WatchSource:0}: Error finding container 6b75993c5d464f2b8dcc9f9116b27b26a70e70ecc92f816bac0d3f3fbba932bc: Status 404 returned error can't find the container with id 6b75993c5d464f2b8dcc9f9116b27b26a70e70ecc92f816bac0d3f3fbba932bc
Sep 06 10:39:22.810669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:22.810628    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7" event={"ID":"19158881-0979-4ec5-97d4-0f433497dca5","Type":"ContainerStarted","Data":"891288ce0615dc38ef907a4eaff34014abe50b04a5c61ed2f19a9e43b52acdd7"}
Sep 06 10:39:22.811102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:22.810689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7" event={"ID":"19158881-0979-4ec5-97d4-0f433497dca5","Type":"ContainerStarted","Data":"6b75993c5d464f2b8dcc9f9116b27b26a70e70ecc92f816bac0d3f3fbba932bc"}
Sep 06 10:39:22.826880 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:22.826823    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7" podStartSLOduration=1.658360789 podStartE2EDuration="1.826802512s" totalImagesPullingTime="168.441723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:39:22.82527955 +0000 UTC m=+545.995129425" watchObservedRunningTime="2026-09-06 10:39:22.826802512 +0000 UTC m=+545.996652386"
Sep 06 10:39:32.039746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:32.039703    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:32.040207 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:32.039770    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:32.051887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:32.051863    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:39:32.867505 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:39:32.867481    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7"
Sep 06 10:40:41.908198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:41.908147    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-754d676c85-js5nj"]
Sep 06 10:40:41.911380 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:41.911363    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:41.914295 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:41.914276    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 06 10:40:41.914405 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:41.914319    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-mjjjs\""
Sep 06 10:40:41.918947 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:41.918925    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754d676c85-js5nj"]
Sep 06 10:40:42.034494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.034464    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbcs\" (UniqueName: \"kubernetes.io/projected/448c56bc-285c-4bf7-aaf9-7577e72f8503-kube-api-access-dqbcs\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.034668 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.034519    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/448c56bc-285c-4bf7-aaf9-7577e72f8503-tls-cert\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.135144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.135111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbcs\" (UniqueName: \"kubernetes.io/projected/448c56bc-285c-4bf7-aaf9-7577e72f8503-kube-api-access-dqbcs\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.135326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.135186    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/448c56bc-285c-4bf7-aaf9-7577e72f8503-tls-cert\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.137727 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.137696    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/448c56bc-285c-4bf7-aaf9-7577e72f8503-tls-cert\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.144599 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.144579    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbcs\" (UniqueName: \"kubernetes.io/projected/448c56bc-285c-4bf7-aaf9-7577e72f8503-kube-api-access-dqbcs\") pod \"authorino-754d676c85-js5nj\" (UID: \"448c56bc-285c-4bf7-aaf9-7577e72f8503\") " pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.221139 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.221056    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-754d676c85-js5nj"
Sep 06 10:40:42.487213 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:42.487187    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-754d676c85-js5nj"]
Sep 06 10:40:42.489701 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:40:42.489657    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod448c56bc_285c_4bf7_aaf9_7577e72f8503.slice/crio-f75a7ffe916b76f405ec5748d37655f9cace538c48cf9d9d97809eb81a5cad61 WatchSource:0}: Error finding container f75a7ffe916b76f405ec5748d37655f9cace538c48cf9d9d97809eb81a5cad61: Status 404 returned error can't find the container with id f75a7ffe916b76f405ec5748d37655f9cace538c48cf9d9d97809eb81a5cad61
Sep 06 10:40:43.162099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:43.162065    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754d676c85-js5nj" event={"ID":"448c56bc-285c-4bf7-aaf9-7577e72f8503","Type":"ContainerStarted","Data":"f75a7ffe916b76f405ec5748d37655f9cace538c48cf9d9d97809eb81a5cad61"}
Sep 06 10:40:44.172053 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:44.172020    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-754d676c85-js5nj" event={"ID":"448c56bc-285c-4bf7-aaf9-7577e72f8503","Type":"ContainerStarted","Data":"980348edb5efac8051c095ba2a86e080bc211b729b61571c3ef61af4b8510d48"}
Sep 06 10:40:44.189899 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:40:44.189852    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-754d676c85-js5nj" podStartSLOduration=2.197610877 podStartE2EDuration="3.189811995s" totalImagesPullingTime="992.201118ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:40:44.188624229 +0000 UTC m=+627.358474114" watchObservedRunningTime="2026-09-06 10:40:44.189811995 +0000 UTC m=+627.359661888"
Sep 06 10:41:02.766062 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:02.766028    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7829-140283554823984"]
Sep 06 10:41:02.769644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:02.769628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:02.771607 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:02.771586    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lwnht\""
Sep 06 10:41:02.775533 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:02.775509    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-140283554823984"]
Sep 06 10:41:02.927831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:02.927801    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw9rp\" (UniqueName: \"kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp\") pod \"test-curl-7829-140283554823984\" (UID: \"5dbfe1a1-b465-4ea5-9d08-9a76476e7762\") " pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:03.029131 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:03.029044    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw9rp\" (UniqueName: \"kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp\") pod \"test-curl-7829-140283554823984\" (UID: \"5dbfe1a1-b465-4ea5-9d08-9a76476e7762\") " pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:03.037585 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:03.037554    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw9rp\" (UniqueName: \"kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp\") pod \"test-curl-7829-140283554823984\" (UID: \"5dbfe1a1-b465-4ea5-9d08-9a76476e7762\") " pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:03.080332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:03.080294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:03.350457 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:03.350431    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dbfe1a1_b465_4ea5_9d08_9a76476e7762.slice/crio-450efad1a1e5a4c1314b7c70d47ff224228be7b3de676bf12853dd1ca6961555 WatchSource:0}: Error finding container 450efad1a1e5a4c1314b7c70d47ff224228be7b3de676bf12853dd1ca6961555: Status 404 returned error can't find the container with id 450efad1a1e5a4c1314b7c70d47ff224228be7b3de676bf12853dd1ca6961555
Sep 06 10:41:03.352911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:03.352893    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7829-140283554823984"]
Sep 06 10:41:04.264476 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:04.264445    2573 generic.go:356] "Generic (PLEG): container finished" podID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" containerID="0b3437c83e4a582b4662b98358ac89ea96d91717ab26c5b3ecabaa0ee19a0d11" exitCode=0
Sep 06 10:41:04.264845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:04.264526    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-140283554823984" event={"ID":"5dbfe1a1-b465-4ea5-9d08-9a76476e7762","Type":"ContainerDied","Data":"0b3437c83e4a582b4662b98358ac89ea96d91717ab26c5b3ecabaa0ee19a0d11"}
Sep 06 10:41:04.264845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:04.264570    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7829-140283554823984" event={"ID":"5dbfe1a1-b465-4ea5-9d08-9a76476e7762","Type":"ContainerStarted","Data":"450efad1a1e5a4c1314b7c70d47ff224228be7b3de676bf12853dd1ca6961555"}
Sep 06 10:41:05.276751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.276718    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:41:05.281381 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.281357    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.283259 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.283236    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-232aeb-serving-cert\""
Sep 06 10:41:05.289825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.289247    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:41:05.350077 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.350048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj75\" (UniqueName: \"kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.350279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.350089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.350279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.350130    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.350279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.350232    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.406811 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.406787    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:05.445912 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.445886    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7829-140283554823984_5dbfe1a1-b465-4ea5-9d08-9a76476e7762/test-curl-7829-140283554823984/0.log"
Sep 06 10:41:05.451106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.451087    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bw9rp\" (UniqueName: \"kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp\") pod \"5dbfe1a1-b465-4ea5-9d08-9a76476e7762\" (UID: \"5dbfe1a1-b465-4ea5-9d08-9a76476e7762\") "
Sep 06 10:41:05.451335 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.451318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj75\" (UniqueName: \"kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.451415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.451372    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.451469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.451454    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.451519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.451492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.451638 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:05.451616    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-serving-cert: secret "maas-api-e2e-inf-a-232aeb-serving-cert" not found
Sep 06 10:41:05.451739 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:05.451686    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls podName:75d26e42-a741-4c6a-8ab6-b2778ae8cdb1 nodeName:}" failed. No retries permitted until 2026-09-06 10:41:05.951665157 +0000 UTC m=+649.121515028 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls") pod "maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" (UID: "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1") : secret "maas-api-e2e-inf-a-232aeb-serving-cert" not found
Sep 06 10:41:05.452362 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.452342    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.453565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.453542    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp" (OuterVolumeSpecName: "kube-api-access-bw9rp") pod "5dbfe1a1-b465-4ea5-9d08-9a76476e7762" (UID: "5dbfe1a1-b465-4ea5-9d08-9a76476e7762"). InnerVolumeSpecName "kube-api-access-bw9rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:05.453895 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.453876    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.459346 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.459319    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj75\" (UniqueName: \"kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.467282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.467257    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7829-140283554823984"]
Sep 06 10:41:05.472962 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.472940    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7829-140283554823984"]
Sep 06 10:41:05.551984 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.551907    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw9rp\" (UniqueName: \"kubernetes.io/projected/5dbfe1a1-b465-4ea5-9d08-9a76476e7762-kube-api-access-bw9rp\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:05.955778 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.955744    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:05.958260 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:05.958236    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") pod \"maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:06.195492 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:06.195447    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:06.279675 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:06.279646    2573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="450efad1a1e5a4c1314b7c70d47ff224228be7b3de676bf12853dd1ca6961555"
Sep 06 10:41:06.280012 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:06.279707    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:06.293147 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:06.293117    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-140283554823984\" is forbidden: User \"system:node:ip-10-0-130-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-119.ec2.internal' and this object" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:06.464436 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:06.464353    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:41:06.467058 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:06.467021    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d26e42_a741_4c6a_8ab6_b2778ae8cdb1.slice/crio-23760c14daddfea27c60603ffe93a29aed37d5b6b922d34ce8516d5b217a5b91 WatchSource:0}: Error finding container 23760c14daddfea27c60603ffe93a29aed37d5b6b922d34ce8516d5b217a5b91: Status 404 returned error can't find the container with id 23760c14daddfea27c60603ffe93a29aed37d5b6b922d34ce8516d5b217a5b91
Sep 06 10:41:07.287772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:07.287719    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" event={"ID":"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1","Type":"ContainerStarted","Data":"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"}
Sep 06 10:41:07.288210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:07.287790    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" event={"ID":"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1","Type":"ContainerStarted","Data":"23760c14daddfea27c60603ffe93a29aed37d5b6b922d34ce8516d5b217a5b91"}
Sep 06 10:41:07.288210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:07.287837    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:07.310178 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:07.310116    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-140283554823984\" is forbidden: User \"system:node:ip-10-0-130-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-119.ec2.internal' and this object" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:07.310732 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:07.310681    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" podStartSLOduration=1.8989591510000001 podStartE2EDuration="2.310667121s" totalImagesPullingTime="411.70797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:07.308427076 +0000 UTC m=+650.478276946" watchObservedRunningTime="2026-09-06 10:41:07.310667121 +0000 UTC m=+650.480516991"
Sep 06 10:41:07.351399 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:07.351361    2573 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7829-140283554823984\" is forbidden: User \"system:node:ip-10-0-130-119.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-119.ec2.internal' and this object" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" pod="opendatahub/test-curl-7829-140283554823984"
Sep 06 10:41:07.352662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:07.352641    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" path="/var/lib/kubelet/pods/5dbfe1a1-b465-4ea5-9d08-9a76476e7762/volumes"
Sep 06 10:41:13.018531 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.018496    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:13.019053 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.019039    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" containerName="test-curl-7829-140283554823984"
Sep 06 10:41:13.019140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.019057    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" containerName="test-curl-7829-140283554823984"
Sep 06 10:41:13.019140 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.019134    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbfe1a1-b465-4ea5-9d08-9a76476e7762" containerName="test-curl-7829-140283554823984"
Sep 06 10:41:13.022421 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.022403    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.025328 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.025295    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e0690faf-serving-cert\""
Sep 06 10:41:13.034210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.034133    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:13.117817 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.117779    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzzg\" (UniqueName: \"kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.117993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.117825    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.117993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.117860    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.117993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.117912    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.218578 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.218545    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.218578 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.218581    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.218806 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.218655    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzzg\" (UniqueName: \"kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.218806 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.218693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.219432 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.219329    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.221156 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.221136    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.221270 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.221226    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.229032 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.229014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzzg\" (UniqueName: \"kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg\") pod \"maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.297837 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.297762    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:41:13.334061 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.334024    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:13.620007 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:13.619982    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:13.622293 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:13.622269    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddf0cd2_bac6_4b8a_9146_42d76f5578aa.slice/crio-211ecf09d4935e9dbb56fc0a84bd937b09474ccbef7496c63d92c37af9dc0f19 WatchSource:0}: Error finding container 211ecf09d4935e9dbb56fc0a84bd937b09474ccbef7496c63d92c37af9dc0f19: Status 404 returned error can't find the container with id 211ecf09d4935e9dbb56fc0a84bd937b09474ccbef7496c63d92c37af9dc0f19
Sep 06 10:41:14.128232 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.128193    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:14.131595 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.131578    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.134656 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.134636    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e0690faf\""
Sep 06 10:41:14.134761 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.134636    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e0690faf-dockercfg-kxcd9\""
Sep 06 10:41:14.147697 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.147662    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:14.227223 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.227139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.227380 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.227302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68sb\" (UniqueName: \"kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.321104 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.321074    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" event={"ID":"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa","Type":"ContainerStarted","Data":"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"}
Sep 06 10:41:14.321282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.321121    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" event={"ID":"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa","Type":"ContainerStarted","Data":"211ecf09d4935e9dbb56fc0a84bd937b09474ccbef7496c63d92c37af9dc0f19"}
Sep 06 10:41:14.321282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.321205    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:14.328399 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.328376    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68sb\" (UniqueName: \"kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.328450 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.328420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.329072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.329055    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.338918 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.338889    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68sb\" (UniqueName: \"kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb\") pod \"payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") " pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.347557 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.347516    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" podStartSLOduration=2.039988207 podStartE2EDuration="2.347504639s" totalImagesPullingTime="307.516432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:14.345622701 +0000 UTC m=+657.515472595" watchObservedRunningTime="2026-09-06 10:41:14.347504639 +0000 UTC m=+657.517354562"
Sep 06 10:41:14.443184 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.443123    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:14.716043 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:14.715961    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:14.718515 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:14.718487    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd83e1d6_c1d0_4a9a_a525_c2b697a29da9.slice/crio-b27d67cc15aae8f531647790be707fbdbf30c82ecd79a561cd3ee685f2ce58b4 WatchSource:0}: Error finding container b27d67cc15aae8f531647790be707fbdbf30c82ecd79a561cd3ee685f2ce58b4: Status 404 returned error can't find the container with id b27d67cc15aae8f531647790be707fbdbf30c82ecd79a561cd3ee685f2ce58b4
Sep 06 10:41:15.169220 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.169184    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:41:15.173325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.173297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.175542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.175521    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-55111c-serving-cert\""
Sep 06 10:41:15.188089 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.188057    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:41:15.238276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.238246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.238459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.238306    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.238534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.238478    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.238534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.238515    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46k9b\" (UniqueName: \"kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.328175 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.328118    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" event={"ID":"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9","Type":"ContainerStarted","Data":"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"}
Sep 06 10:41:15.328379 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.328197    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" event={"ID":"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9","Type":"ContainerStarted","Data":"b27d67cc15aae8f531647790be707fbdbf30c82ecd79a561cd3ee685f2ce58b4"}
Sep 06 10:41:15.328379 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.328306    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:15.339060 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.339033    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.339134 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.339063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46k9b\" (UniqueName: \"kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.339134 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.339085    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.339246 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.339210    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.339762 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.339742    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.341623 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.341603    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.341851 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.341832    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.348276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.348231    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" podStartSLOduration=1.348214434 podStartE2EDuration="1.348214434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:15.345754076 +0000 UTC m=+658.515603950" watchObservedRunningTime="2026-09-06 10:41:15.348214434 +0000 UTC m=+658.518064305"
Sep 06 10:41:15.350535 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.350511    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46k9b\" (UniqueName: \"kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b\") pod \"maas-api-e2e-autores-55111c-58b49cfb45-dhjz2\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.485500 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.485422    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:15.760506 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.760480    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:41:15.761790 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:15.761764    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c81057e_1ae4_423a_9739_f6cb28d2bc8a.slice/crio-b4b359cd083e3e72f16db42ff4ce0ab44428987772f14704a8efb0d0417bf9eb WatchSource:0}: Error finding container b4b359cd083e3e72f16db42ff4ce0ab44428987772f14704a8efb0d0417bf9eb: Status 404 returned error can't find the container with id b4b359cd083e3e72f16db42ff4ce0ab44428987772f14704a8efb0d0417bf9eb
Sep 06 10:41:15.763360 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:15.763341    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:41:16.334398 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:16.334361    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" event={"ID":"2c81057e-1ae4-423a-9739-f6cb28d2bc8a","Type":"ContainerStarted","Data":"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"}
Sep 06 10:41:16.334825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:16.334418    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" event={"ID":"2c81057e-1ae4-423a-9739-f6cb28d2bc8a","Type":"ContainerStarted","Data":"b4b359cd083e3e72f16db42ff4ce0ab44428987772f14704a8efb0d0417bf9eb"}
Sep 06 10:41:16.334825 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:16.334519    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:16.352383 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:16.352338    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" podStartSLOduration=0.940527069 podStartE2EDuration="1.352321278s" totalImagesPullingTime="411.794209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:16.350989097 +0000 UTC m=+659.520838981" watchObservedRunningTime="2026-09-06 10:41:16.352321278 +0000 UTC m=+659.522171192"
Sep 06 10:41:20.334223 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:20.334183    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:22.347646 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:22.347610    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:41:26.336773 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:26.336737    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:33.546845 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.546813    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:41:33.552470 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.552446    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.554591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.554564    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-636b04-serving-cert\""
Sep 06 10:41:33.561041 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.559828    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:41:33.723636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.723599    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.723636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.723637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.723831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.723733    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.723831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.723760    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtvk9\" (UniqueName: \"kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.825127 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.825041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.825127 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.825086    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtvk9\" (UniqueName: \"kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.825362 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.825244    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.825362 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.825271    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.825881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.825855    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.828019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.827997    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.828019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.828007    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.833119 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.833097    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtvk9\" (UniqueName: \"kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9\") pod \"maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:33.866113 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:33.866085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:34.145075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:34.145053    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:41:34.147122 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:34.147091    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3f86be_ea8e_48f8_a682_a26cf2a922d8.slice/crio-0f660b37a9ebb67d54e1580c6336c1e1347f850459d3c77a725cc2f9b5379832 WatchSource:0}: Error finding container 0f660b37a9ebb67d54e1580c6336c1e1347f850459d3c77a725cc2f9b5379832: Status 404 returned error can't find the container with id 0f660b37a9ebb67d54e1580c6336c1e1347f850459d3c77a725cc2f9b5379832
Sep 06 10:41:34.431606 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:34.431503    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" event={"ID":"8b3f86be-ea8e-48f8-a682-a26cf2a922d8","Type":"ContainerStarted","Data":"0f660b37a9ebb67d54e1580c6336c1e1347f850459d3c77a725cc2f9b5379832"}
Sep 06 10:41:35.438090 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.438053    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" event={"ID":"8b3f86be-ea8e-48f8-a682-a26cf2a922d8","Type":"ContainerStarted","Data":"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"}
Sep 06 10:41:35.438536 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.438230    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:35.460689 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.460633    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" podStartSLOduration=2.144170035 podStartE2EDuration="2.460613919s" totalImagesPullingTime="316.443884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:35.459302148 +0000 UTC m=+678.629152018" watchObservedRunningTime="2026-09-06 10:41:35.460613919 +0000 UTC m=+678.630463788"
Sep 06 10:41:35.511696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.511659    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:41:35.515568 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.515548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.517521 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.517505    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-511895d8-kserve-self-signed-certs\""
Sep 06 10:41:35.524235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.524213    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:41:35.644876 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.644842    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.644876 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.644883    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.645147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.644900    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt49f\" (UniqueName: \"kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.645147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.644965    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.645147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.645057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.645147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.645129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746399    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746452    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt49f\" (UniqueName: \"kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746559    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746616    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.746945 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746924    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.747004 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.746983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.747057 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.747025    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.748861 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.748843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.749093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.749077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.754960 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.754940    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt49f\" (UniqueName: \"kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f\") pod \"e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") " pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:35.825808 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:35.825771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:36.104139 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:36.103983    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:41:36.108828 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:36.108794    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b84f684_020e_45b5_a2ba_36f243efc180.slice/crio-13da7be5a199a26cbc4d3621afda9086ce519e5acc12959e065019ba4b4bb741 WatchSource:0}: Error finding container 13da7be5a199a26cbc4d3621afda9086ce519e5acc12959e065019ba4b4bb741: Status 404 returned error can't find the container with id 13da7be5a199a26cbc4d3621afda9086ce519e5acc12959e065019ba4b4bb741
Sep 06 10:41:36.444660 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:36.444617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" event={"ID":"4b84f684-020e-45b5-a2ba-36f243efc180","Type":"ContainerStarted","Data":"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"}
Sep 06 10:41:36.445051 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:36.444670    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" event={"ID":"4b84f684-020e-45b5-a2ba-36f243efc180","Type":"ContainerStarted","Data":"13da7be5a199a26cbc4d3621afda9086ce519e5acc12959e065019ba4b4bb741"}
Sep 06 10:41:36.461619 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:36.461574    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" podStartSLOduration=1.461557702 podStartE2EDuration="1.461557702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:36.459972523 +0000 UTC m=+679.629822395" watchObservedRunningTime="2026-09-06 10:41:36.461557702 +0000 UTC m=+679.631407570"
Sep 06 10:41:41.450911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:41.450881    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:41:45.826341 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:45.826296    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:45.826817 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:45.826363    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:45.838520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:45.838498    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:46.503006 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:46.502977    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:41:53.776078 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:53.775889    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:53.776590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:53.776149    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" podUID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" containerName="maas-api" containerID="cri-o://bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc" gracePeriod=30
Sep 06 10:41:53.835407 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:53.835373    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:53.835667 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:53.835621    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" podUID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" containerName="payload-processing" containerID="cri-o://da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90" gracePeriod=30
Sep 06 10:41:54.082517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.082491    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:54.116215 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.116186    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:54.130715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.130686    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs\") pod \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") "
Sep 06 10:41:54.130875 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.130727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls\") pod \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") "
Sep 06 10:41:54.130875 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.130768    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdzzg\" (UniqueName: \"kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg\") pod \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") "
Sep 06 10:41:54.130994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.130941    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca\") pod \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\" (UID: \"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa\") "
Sep 06 10:41:54.131703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.131654    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" (UID: "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:41:54.134494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.134456    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg" (OuterVolumeSpecName: "kube-api-access-sdzzg") pod "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" (UID: "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa"). InnerVolumeSpecName "kube-api-access-sdzzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:54.134626 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.134497    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" (UID: "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:41:54.134744 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.134612    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" (UID: "0ddf0cd2-bac6-4b8a-9146-42d76f5578aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:41:54.232035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232000    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume\") pod \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") "
Sep 06 10:41:54.232228 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232043    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t68sb\" (UniqueName: \"kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb\") pod \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\" (UID: \"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9\") "
Sep 06 10:41:54.232278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232240    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.232278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232252    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.232278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232261    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzzg\" (UniqueName: \"kubernetes.io/projected/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-kube-api-access-sdzzg\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.232278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232270    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.232415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.232393    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" (UID: "bd83e1d6-c1d0-4a9a-a525-c2b697a29da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:41:54.234357 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.234335    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb" (OuterVolumeSpecName: "kube-api-access-t68sb") pod "bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" (UID: "bd83e1d6-c1d0-4a9a-a525-c2b697a29da9"). InnerVolumeSpecName "kube-api-access-t68sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:41:54.333358 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.333275    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.333358 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.333302    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68sb\" (UniqueName: \"kubernetes.io/projected/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9-kube-api-access-t68sb\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:41:54.540929 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.540895    2573 generic.go:356] "Generic (PLEG): container finished" podID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" containerID="da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90" exitCode=0
Sep 06 10:41:54.541099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.540941    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"
Sep 06 10:41:54.541099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.540981    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" event={"ID":"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9","Type":"ContainerDied","Data":"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"}
Sep 06 10:41:54.541099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.541041    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5" event={"ID":"bd83e1d6-c1d0-4a9a-a525-c2b697a29da9","Type":"ContainerDied","Data":"b27d67cc15aae8f531647790be707fbdbf30c82ecd79a561cd3ee685f2ce58b4"}
Sep 06 10:41:54.541099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.541075    2573 scope.go:117] "RemoveContainer" containerID="da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"
Sep 06 10:41:54.542482 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.542460    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" containerID="bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc" exitCode=0
Sep 06 10:41:54.542550 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.542497    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"
Sep 06 10:41:54.542550 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.542506    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" event={"ID":"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa","Type":"ContainerDied","Data":"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"}
Sep 06 10:41:54.542550 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.542541    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb" event={"ID":"0ddf0cd2-bac6-4b8a-9146-42d76f5578aa","Type":"ContainerDied","Data":"211ecf09d4935e9dbb56fc0a84bd937b09474ccbef7496c63d92c37af9dc0f19"}
Sep 06 10:41:54.551050 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.551035    2573 scope.go:117] "RemoveContainer" containerID="da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"
Sep 06 10:41:54.551360 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:54.551341    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90\": container with ID starting with da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90 not found: ID does not exist" containerID="da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"
Sep 06 10:41:54.551429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.551369    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90"} err="failed to get container status \"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90\": rpc error: code = NotFound desc = could not find container \"da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90\": container with ID starting with da65c3e620b92ae7d4795813220f317c16f245cf9806ff322122cf2a9cb89f90 not found: ID does not exist"
Sep 06 10:41:54.551429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.551386    2573 scope.go:117] "RemoveContainer" containerID="bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"
Sep 06 10:41:54.559306 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.559290    2573 scope.go:117] "RemoveContainer" containerID="bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"
Sep 06 10:41:54.559604 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:41:54.559584    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc\": container with ID starting with bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc not found: ID does not exist" containerID="bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"
Sep 06 10:41:54.559686 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.559623    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc"} err="failed to get container status \"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc\": rpc error: code = NotFound desc = could not find container \"bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc\": container with ID starting with bd839599bf2b21620d7717b152c92a70cc9c6220f34d003ea3f21d5d2cdab7fc not found: ID does not exist"
Sep 06 10:41:54.566552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.566526    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:54.575002 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.574979    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e0690faf-6f674478b7-r6vzb"]
Sep 06 10:41:54.591675 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.591609    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:54.602180 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:54.602139    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e0690faf-7746444d77-gpnv5"]
Sep 06 10:41:55.353716 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:55.353686    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" path="/var/lib/kubelet/pods/0ddf0cd2-bac6-4b8a-9146-42d76f5578aa/volumes"
Sep 06 10:41:55.354084 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:55.354049    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" path="/var/lib/kubelet/pods/bd83e1d6-c1d0-4a9a-a525-c2b697a29da9/volumes"
Sep 06 10:41:58.265603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.265563    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:41:58.266113 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266092    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" containerName="maas-api"
Sep 06 10:41:58.266234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266117    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" containerName="maas-api"
Sep 06 10:41:58.266234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266135    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" containerName="payload-processing"
Sep 06 10:41:58.266234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266143    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" containerName="payload-processing"
Sep 06 10:41:58.266383 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266281    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ddf0cd2-bac6-4b8a-9146-42d76f5578aa" containerName="maas-api"
Sep 06 10:41:58.266383 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.266297    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd83e1d6-c1d0-4a9a-a525-c2b697a29da9" containerName="payload-processing"
Sep 06 10:41:58.270831 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.270809    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.272829 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.272807    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-2dae43f5-serving-cert\""
Sep 06 10:41:58.281197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.281155    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:41:58.372016 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.371983    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.372214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.372140    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.372268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.372243    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwbxw\" (UniqueName: \"kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.372306 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.372267    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.472944 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.472914    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.473122 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.473023    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.473122 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.473075    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwbxw\" (UniqueName: \"kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.473122 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.473100    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.473661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.473625    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.475770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.475745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.475830 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.475745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.481143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.481112    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwbxw\" (UniqueName: \"kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw\") pod \"maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.581894 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.581812    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:58.864150 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:58.864110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:41:58.866558 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:58.866528    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8db936dc_5d03_441d_9d80_4ebc74e8630d.slice/crio-ff738a5b16344ef7af4ba1c775d9d98cf5d98cb25175b62bef35e0b75e00c53d WatchSource:0}: Error finding container ff738a5b16344ef7af4ba1c775d9d98cf5d98cb25175b62bef35e0b75e00c53d: Status 404 returned error can't find the container with id ff738a5b16344ef7af4ba1c775d9d98cf5d98cb25175b62bef35e0b75e00c53d
Sep 06 10:41:59.071813 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.071777    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:41:59.078305 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.078282    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.080563 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.080533    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2dae43f5-dockercfg-tdjst\""
Sep 06 10:41:59.080875 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.080857    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2dae43f5\""
Sep 06 10:41:59.085743 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.085719    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:41:59.179628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.179596    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2l6z\" (UniqueName: \"kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.179801 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.179648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.280487 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.280446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.280868 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.280579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2l6z\" (UniqueName: \"kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.281246 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.281222    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.290804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.290772    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2l6z\" (UniqueName: \"kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z\") pod \"payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") " pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.390992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.390958    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:41:59.572677 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.572637    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" event={"ID":"8db936dc-5d03-441d-9d80-4ebc74e8630d","Type":"ContainerStarted","Data":"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"}
Sep 06 10:41:59.572865 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.572692    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" event={"ID":"8db936dc-5d03-441d-9d80-4ebc74e8630d","Type":"ContainerStarted","Data":"ff738a5b16344ef7af4ba1c775d9d98cf5d98cb25175b62bef35e0b75e00c53d"}
Sep 06 10:41:59.572865 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.572752    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:41:59.588717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.588668    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" podStartSLOduration=1.26753312 podStartE2EDuration="1.588649905s" totalImagesPullingTime="321.116785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:41:59.587439507 +0000 UTC m=+702.757289379" watchObservedRunningTime="2026-09-06 10:41:59.588649905 +0000 UTC m=+702.758499776"
Sep 06 10:41:59.666046 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:41:59.665992    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:41:59.668095 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:41:59.668066    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffef54e8_80ee_4df8_b8dc_e79439aedfb3.slice/crio-6a001476b22fd0d0263945eea5159956adc0d3831ad1fecb19c8232a6de38c31 WatchSource:0}: Error finding container 6a001476b22fd0d0263945eea5159956adc0d3831ad1fecb19c8232a6de38c31: Status 404 returned error can't find the container with id 6a001476b22fd0d0263945eea5159956adc0d3831ad1fecb19c8232a6de38c31
Sep 06 10:42:00.579524 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:00.579485    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" event={"ID":"ffef54e8-80ee-4df8-b8dc-e79439aedfb3","Type":"ContainerStarted","Data":"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"}
Sep 06 10:42:00.580004 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:00.579557    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" event={"ID":"ffef54e8-80ee-4df8-b8dc-e79439aedfb3","Type":"ContainerStarted","Data":"6a001476b22fd0d0263945eea5159956adc0d3831ad1fecb19c8232a6de38c31"}
Sep 06 10:42:00.580004 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:00.579770    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:42:00.597684 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:00.597621    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" podStartSLOduration=1.5976009549999999 podStartE2EDuration="1.597600955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:00.597304099 +0000 UTC m=+703.767153965" watchObservedRunningTime="2026-09-06 10:42:00.597600955 +0000 UTC m=+703.767450826"
Sep 06 10:42:05.584849 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:05.584814    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:42:11.587152 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:11.587118    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:42:29.048929 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.048893    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:42:29.049465 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.049112    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" podUID="8db936dc-5d03-441d-9d80-4ebc74e8630d" containerName="maas-api" containerID="cri-o://d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723" gracePeriod=30
Sep 06 10:42:29.107612 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.107579    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:42:29.107862 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.107824    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" podUID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" containerName="payload-processing" containerID="cri-o://3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676" gracePeriod=30
Sep 06 10:42:29.380139 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.380112    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:42:29.383277 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.383195    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:42:29.481252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481214    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2l6z\" (UniqueName: \"kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z\") pod \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") "
Sep 06 10:42:29.481445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume\") pod \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\" (UID: \"ffef54e8-80ee-4df8-b8dc-e79439aedfb3\") "
Sep 06 10:42:29.481445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481299    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls\") pod \"8db936dc-5d03-441d-9d80-4ebc74e8630d\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") "
Sep 06 10:42:29.481445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481342    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwbxw\" (UniqueName: \"kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw\") pod \"8db936dc-5d03-441d-9d80-4ebc74e8630d\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") "
Sep 06 10:42:29.481445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481401    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca\") pod \"8db936dc-5d03-441d-9d80-4ebc74e8630d\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") "
Sep 06 10:42:29.481445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481422    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs\") pod \"8db936dc-5d03-441d-9d80-4ebc74e8630d\" (UID: \"8db936dc-5d03-441d-9d80-4ebc74e8630d\") "
Sep 06 10:42:29.481728 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481697    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ffef54e8-80ee-4df8-b8dc-e79439aedfb3" (UID: "ffef54e8-80ee-4df8-b8dc-e79439aedfb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:42:29.481786 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.481728    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8db936dc-5d03-441d-9d80-4ebc74e8630d" (UID: "8db936dc-5d03-441d-9d80-4ebc74e8630d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:42:29.483664 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.483635    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z" (OuterVolumeSpecName: "kube-api-access-p2l6z") pod "ffef54e8-80ee-4df8-b8dc-e79439aedfb3" (UID: "ffef54e8-80ee-4df8-b8dc-e79439aedfb3"). InnerVolumeSpecName "kube-api-access-p2l6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:42:29.483664 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.483647    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw" (OuterVolumeSpecName: "kube-api-access-zwbxw") pod "8db936dc-5d03-441d-9d80-4ebc74e8630d" (UID: "8db936dc-5d03-441d-9d80-4ebc74e8630d"). InnerVolumeSpecName "kube-api-access-zwbxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:42:29.483664 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.483661    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8db936dc-5d03-441d-9d80-4ebc74e8630d" (UID: "8db936dc-5d03-441d-9d80-4ebc74e8630d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:42:29.483829 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.483694    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8db936dc-5d03-441d-9d80-4ebc74e8630d" (UID: "8db936dc-5d03-441d-9d80-4ebc74e8630d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582650    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582685    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582695    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwbxw\" (UniqueName: \"kubernetes.io/projected/8db936dc-5d03-441d-9d80-4ebc74e8630d-kube-api-access-zwbxw\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582704    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8db936dc-5d03-441d-9d80-4ebc74e8630d-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582714    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8db936dc-5d03-441d-9d80-4ebc74e8630d-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.582737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.582722    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2l6z\" (UniqueName: \"kubernetes.io/projected/ffef54e8-80ee-4df8-b8dc-e79439aedfb3-kube-api-access-p2l6z\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:42:29.720533 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.720497    2573 generic.go:356] "Generic (PLEG): container finished" podID="8db936dc-5d03-441d-9d80-4ebc74e8630d" containerID="d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723" exitCode=0
Sep 06 10:42:29.720712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.720545    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"
Sep 06 10:42:29.720712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.720567    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" event={"ID":"8db936dc-5d03-441d-9d80-4ebc74e8630d","Type":"ContainerDied","Data":"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"}
Sep 06 10:42:29.720712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.720615    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm" event={"ID":"8db936dc-5d03-441d-9d80-4ebc74e8630d","Type":"ContainerDied","Data":"ff738a5b16344ef7af4ba1c775d9d98cf5d98cb25175b62bef35e0b75e00c53d"}
Sep 06 10:42:29.720712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.720636    2573 scope.go:117] "RemoveContainer" containerID="d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"
Sep 06 10:42:29.730191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.722292    2573 generic.go:356] "Generic (PLEG): container finished" podID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" containerID="3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676" exitCode=0
Sep 06 10:42:29.730191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.722384    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" event={"ID":"ffef54e8-80ee-4df8-b8dc-e79439aedfb3","Type":"ContainerDied","Data":"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"}
Sep 06 10:42:29.730191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.722454    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx" event={"ID":"ffef54e8-80ee-4df8-b8dc-e79439aedfb3","Type":"ContainerDied","Data":"6a001476b22fd0d0263945eea5159956adc0d3831ad1fecb19c8232a6de38c31"}
Sep 06 10:42:29.730191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.722541    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"
Sep 06 10:42:29.735971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.735952    2573 scope.go:117] "RemoveContainer" containerID="d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"
Sep 06 10:42:29.736313 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:42:29.736290    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723\": container with ID starting with d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723 not found: ID does not exist" containerID="d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"
Sep 06 10:42:29.736404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.736333    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723"} err="failed to get container status \"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723\": rpc error: code = NotFound desc = could not find container \"d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723\": container with ID starting with d25ae05b8826ff158ea2fb0fc0023fc0bb24712f42d913672b6bdd68bd6f0723 not found: ID does not exist"
Sep 06 10:42:29.736404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.736356    2573 scope.go:117] "RemoveContainer" containerID="3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"
Sep 06 10:42:29.744291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.744275    2573 scope.go:117] "RemoveContainer" containerID="3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"
Sep 06 10:42:29.744673 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:42:29.744649    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676\": container with ID starting with 3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676 not found: ID does not exist" containerID="3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"
Sep 06 10:42:29.744796 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.744680    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676"} err="failed to get container status \"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676\": rpc error: code = NotFound desc = could not find container \"3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676\": container with ID starting with 3880096ac8d4b0dae5e32af059037fe6f06a32f8190a8cc8c73badfb459cd676 not found: ID does not exist"
Sep 06 10:42:29.747227 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.747204    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:42:29.756242 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.756220    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-2dae43f5-ccc9bf4b9-vbkcm"]
Sep 06 10:42:29.767433 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.767405    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:42:29.773909 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:29.773882    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2dae43f5-668bfc8f8-v2nkx"]
Sep 06 10:42:31.352412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:31.352382    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db936dc-5d03-441d-9d80-4ebc74e8630d" path="/var/lib/kubelet/pods/8db936dc-5d03-441d-9d80-4ebc74e8630d/volumes"
Sep 06 10:42:31.352774 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:31.352744    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" path="/var/lib/kubelet/pods/ffef54e8-80ee-4df8-b8dc-e79439aedfb3/volumes"
Sep 06 10:42:33.424315 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424280    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:42:33.424828 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424808    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" containerName="payload-processing"
Sep 06 10:42:33.424913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424832    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" containerName="payload-processing"
Sep 06 10:42:33.424913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424867    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db936dc-5d03-441d-9d80-4ebc74e8630d" containerName="maas-api"
Sep 06 10:42:33.424913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424873    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db936dc-5d03-441d-9d80-4ebc74e8630d" containerName="maas-api"
Sep 06 10:42:33.425120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424972    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffef54e8-80ee-4df8-b8dc-e79439aedfb3" containerName="payload-processing"
Sep 06 10:42:33.425120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.424985    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db936dc-5d03-441d-9d80-4ebc74e8630d" containerName="maas-api"
Sep 06 10:42:33.428239 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.428215    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.430473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.430447    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2b45b43b-serving-cert\""
Sep 06 10:42:33.439489 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.439461    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:42:33.525627 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.525595    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.525804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.525647    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7gfm\" (UniqueName: \"kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.525804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.525733    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.525804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.525768    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.626641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.626601    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.626641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.626644    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.627019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.626691    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.627019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.626731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7gfm\" (UniqueName: \"kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.627019 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:42:33.626848    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-serving-cert: secret "maas-api-e2e-ait-2b45b43b-serving-cert" not found
Sep 06 10:42:33.627019 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:42:33.626927    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls podName:4a98e6a6-51d8-47f8-834d-c71b2cbf25d9 nodeName:}" failed. No retries permitted until 2026-09-06 10:42:34.126907355 +0000 UTC m=+737.296757211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls") pod "maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" (UID: "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9") : secret "maas-api-e2e-ait-2b45b43b-serving-cert" not found
Sep 06 10:42:33.627373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.627354    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.629117 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.629094    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:33.635135 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:33.635109    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7gfm\" (UniqueName: \"kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:34.090982 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.090950    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:42:34.094692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.094673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.099707 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.099682    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b45b43b-dockercfg-gbwc8\""
Sep 06 10:42:34.099843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.099719    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b45b43b\""
Sep 06 10:42:34.114215 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.114184    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:42:34.131570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.131530    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:34.134219 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.134192    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") pod \"maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:34.232784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.232751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.232961 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.232835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzxm4\" (UniqueName: \"kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.334073 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.334031    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzxm4\" (UniqueName: \"kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.334313 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.334194    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.335276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.335232    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.339889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.339870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:34.345779 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.345756    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzxm4\" (UniqueName: \"kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4\") pod \"payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") " pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.404682 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.404626    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:34.615192 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.615133    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:42:34.617705 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:42:34.617680    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a98e6a6_51d8_47f8_834d_c71b2cbf25d9.slice/crio-5548824103b548c48c6f23f037319757799884a6bb5d8702a84bb1dbd41a3014 WatchSource:0}: Error finding container 5548824103b548c48c6f23f037319757799884a6bb5d8702a84bb1dbd41a3014: Status 404 returned error can't find the container with id 5548824103b548c48c6f23f037319757799884a6bb5d8702a84bb1dbd41a3014
Sep 06 10:42:34.688226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.688192    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:42:34.689671 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:42:34.689641    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a853ba_5aa7_4ea3_9485_bfecf3c0d065.slice/crio-44ec2f409ed43d2073be8c2ae48baf4dd6e93b1a0692472d17aa6af1a6fce1c8 WatchSource:0}: Error finding container 44ec2f409ed43d2073be8c2ae48baf4dd6e93b1a0692472d17aa6af1a6fce1c8: Status 404 returned error can't find the container with id 44ec2f409ed43d2073be8c2ae48baf4dd6e93b1a0692472d17aa6af1a6fce1c8
Sep 06 10:42:34.748057 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.748014    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" event={"ID":"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065","Type":"ContainerStarted","Data":"44ec2f409ed43d2073be8c2ae48baf4dd6e93b1a0692472d17aa6af1a6fce1c8"}
Sep 06 10:42:34.749290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:34.749263    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" event={"ID":"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9","Type":"ContainerStarted","Data":"5548824103b548c48c6f23f037319757799884a6bb5d8702a84bb1dbd41a3014"}
Sep 06 10:42:35.755948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.755907    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" event={"ID":"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065","Type":"ContainerStarted","Data":"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"}
Sep 06 10:42:35.756457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.756081    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:35.757622 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.757599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" event={"ID":"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9","Type":"ContainerStarted","Data":"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"}
Sep 06 10:42:35.757744 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.757672    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:35.771272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.771219    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" podStartSLOduration=1.771203753 podStartE2EDuration="1.771203753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:35.770605868 +0000 UTC m=+738.940455740" watchObservedRunningTime="2026-09-06 10:42:35.771203753 +0000 UTC m=+738.941053623"
Sep 06 10:42:35.786268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:35.786219    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" podStartSLOduration=2.412729601 podStartE2EDuration="2.78620272s" totalImagesPullingTime="373.473119ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:35.785563378 +0000 UTC m=+738.955413249" watchObservedRunningTime="2026-09-06 10:42:35.78620272 +0000 UTC m=+738.956052590"
Sep 06 10:42:41.768290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:41.768257    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:42:46.765984 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:46.765948    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:42:55.123079 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.123045    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:42:55.127293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.127271    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.130558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.130528    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-55111c\"/\"openshift-service-ca.crt\""
Sep 06 10:42:55.130558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.130549    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-55111c\"/\"kube-root-ca.crt\""
Sep 06 10:42:55.130756 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.130567    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-55111c\"/\"autores-model-55111c-kserve-self-signed-certs\""
Sep 06 10:42:55.130756 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.130574    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-55111c\"/\"default-dockercfg-wp6ww\""
Sep 06 10:42:55.131472 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.131472 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.131638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131537    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.131638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131589    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.131638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131619    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.131823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.131642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xdd6\" (UniqueName: \"kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.135263 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.135189    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:42:55.232398 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232361    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232398 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232623 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232623 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232621    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232724 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232643    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232724 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232671    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xdd6\" (UniqueName: \"kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232897 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.232897 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232890    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.233040 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.232965    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.235013 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.234986    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.235504 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.235485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.239683 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.239659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xdd6\" (UniqueName: \"kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6\") pod \"autores-model-55111c-kserve-5d57b789d7-82489\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") " pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.437326 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.437288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:42:55.712275 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.712247    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:42:55.714747 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:42:55.714717    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde05c560_55a9_4ce4_895a_ebd40885de33.slice/crio-a770b92dedf553046d088f6e347124329d6d7b58a0b6e8f95dfbd81d2d60db63 WatchSource:0}: Error finding container a770b92dedf553046d088f6e347124329d6d7b58a0b6e8f95dfbd81d2d60db63: Status 404 returned error can't find the container with id a770b92dedf553046d088f6e347124329d6d7b58a0b6e8f95dfbd81d2d60db63
Sep 06 10:42:55.869381 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.869338    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" event={"ID":"de05c560-55a9-4ce4-895a-ebd40885de33","Type":"ContainerStarted","Data":"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"}
Sep 06 10:42:55.869547 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.869390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" event={"ID":"de05c560-55a9-4ce4-895a-ebd40885de33","Type":"ContainerStarted","Data":"a770b92dedf553046d088f6e347124329d6d7b58a0b6e8f95dfbd81d2d60db63"}
Sep 06 10:42:55.885538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:42:55.885492    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" podStartSLOduration=0.885476542 podStartE2EDuration="885.476542ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:42:55.88354998 +0000 UTC m=+759.053399853" watchObservedRunningTime="2026-09-06 10:42:55.885476542 +0000 UTC m=+759.055326442"
Sep 06 10:43:04.176297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.176198    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:43:04.176794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.176495    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" podUID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" containerName="maas-api" containerID="cri-o://2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022" gracePeriod=30
Sep 06 10:43:04.255240 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.255199    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:43:04.255447 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.255410    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" podUID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" containerName="payload-processing" containerID="cri-o://a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9" gracePeriod=30
Sep 06 10:43:04.505392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.505366    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:43:04.537325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.537290    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:43:04.620287 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620252    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7gfm\" (UniqueName: \"kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm\") pod \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") "
Sep 06 10:43:04.620452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620313    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume\") pod \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") "
Sep 06 10:43:04.620452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620340    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") pod \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") "
Sep 06 10:43:04.620452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620378    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca\") pod \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") "
Sep 06 10:43:04.620452 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620422    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzxm4\" (UniqueName: \"kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4\") pod \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\" (UID: \"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065\") "
Sep 06 10:43:04.620624 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620468    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs\") pod \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\" (UID: \"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9\") "
Sep 06 10:43:04.620807 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620764    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" (UID: "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:43:04.620927 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.620783    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" (UID: "e1a853ba-5aa7-4ea3-9485-bfecf3c0d065"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:43:04.622749 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.622718    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4" (OuterVolumeSpecName: "kube-api-access-lzxm4") pod "e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" (UID: "e1a853ba-5aa7-4ea3-9485-bfecf3c0d065"). InnerVolumeSpecName "kube-api-access-lzxm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:04.622749 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.622725    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" (UID: "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:04.622749 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.622736    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm" (OuterVolumeSpecName: "kube-api-access-x7gfm") pod "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" (UID: "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9"). InnerVolumeSpecName "kube-api-access-x7gfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:04.622915 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.622760    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" (UID: "4a98e6a6-51d8-47f8-834d-c71b2cbf25d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722147    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7gfm\" (UniqueName: \"kubernetes.io/projected/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-kube-api-access-x7gfm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722212    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722222    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722231    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722239    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzxm4\" (UniqueName: \"kubernetes.io/projected/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065-kube-api-access-lzxm4\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.722272 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.722248    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:04.916206 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.916142    2573 generic.go:356] "Generic (PLEG): container finished" podID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" containerID="a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9" exitCode=0
Sep 06 10:43:04.916206 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.916204    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"
Sep 06 10:43:04.916448 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.916228    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" event={"ID":"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065","Type":"ContainerDied","Data":"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"}
Sep 06 10:43:04.916448 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.916279    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx" event={"ID":"e1a853ba-5aa7-4ea3-9485-bfecf3c0d065","Type":"ContainerDied","Data":"44ec2f409ed43d2073be8c2ae48baf4dd6e93b1a0692472d17aa6af1a6fce1c8"}
Sep 06 10:43:04.916448 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.916301    2573 scope.go:117] "RemoveContainer" containerID="a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"
Sep 06 10:43:04.917659 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.917637    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" containerID="2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022" exitCode=0
Sep 06 10:43:04.917743 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.917664    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" event={"ID":"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9","Type":"ContainerDied","Data":"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"}
Sep 06 10:43:04.917743 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.917689    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"
Sep 06 10:43:04.917743 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.917714    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld" event={"ID":"4a98e6a6-51d8-47f8-834d-c71b2cbf25d9","Type":"ContainerDied","Data":"5548824103b548c48c6f23f037319757799884a6bb5d8702a84bb1dbd41a3014"}
Sep 06 10:43:04.925242 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.925227    2573 scope.go:117] "RemoveContainer" containerID="a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"
Sep 06 10:43:04.925570 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:04.925545    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9\": container with ID starting with a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9 not found: ID does not exist" containerID="a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"
Sep 06 10:43:04.925685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.925576    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9"} err="failed to get container status \"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9\": rpc error: code = NotFound desc = could not find container \"a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9\": container with ID starting with a26938a0ec5904c34a5cbe6792076bde8c12c216b394c27c3342305165a034f9 not found: ID does not exist"
Sep 06 10:43:04.925685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.925593    2573 scope.go:117] "RemoveContainer" containerID="2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"
Sep 06 10:43:04.933999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.933982    2573 scope.go:117] "RemoveContainer" containerID="2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"
Sep 06 10:43:04.934284 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:04.934264    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022\": container with ID starting with 2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022 not found: ID does not exist" containerID="2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"
Sep 06 10:43:04.934353 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.934290    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022"} err="failed to get container status \"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022\": rpc error: code = NotFound desc = could not find container \"2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022\": container with ID starting with 2238ce82d23d1baa3eaeaaa5c011d01698a550b60d3d2d9588854b93c1589022 not found: ID does not exist"
Sep 06 10:43:04.939150 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.939126    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:43:04.942024 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.942002    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2b45b43b-677b5d56c6-mk9ld"]
Sep 06 10:43:04.949255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.949234    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:43:04.950868 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:04.950850    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b45b43b-745d8b5779-zr2nx"]
Sep 06 10:43:05.353580 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.353543    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" path="/var/lib/kubelet/pods/4a98e6a6-51d8-47f8-834d-c71b2cbf25d9/volumes"
Sep 06 10:43:05.353952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.353898    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" path="/var/lib/kubelet/pods/e1a853ba-5aa7-4ea3-9485-bfecf3c0d065/volumes"
Sep 06 10:43:05.438028 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.437994    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:05.438207 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.438041    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:05.450509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.450420    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:05.936097 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:05.936070    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:09.647803 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.647769    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:09.648438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648413    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" containerName="maas-api"
Sep 06 10:43:09.648438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648437    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" containerName="maas-api"
Sep 06 10:43:09.648591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648461    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" containerName="payload-processing"
Sep 06 10:43:09.648591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648471    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" containerName="payload-processing"
Sep 06 10:43:09.648591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648569    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a98e6a6-51d8-47f8-834d-c71b2cbf25d9" containerName="maas-api"
Sep 06 10:43:09.648591 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.648586    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1a853ba-5aa7-4ea3-9485-bfecf3c0d065" containerName="payload-processing"
Sep 06 10:43:09.653963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.653937    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.656231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.656207    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-02ec029f-serving-cert\""
Sep 06 10:43:09.663228 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.661891    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:09.770221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.770155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.770391 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.770224    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db8p9\" (UniqueName: \"kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.770391 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.770278    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.770391 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.770313    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.871772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.871731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.871970 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.871777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db8p9\" (UniqueName: \"kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.871970 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.871807    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.871970 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.871833    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.871970 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:09.871934    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-serving-cert: secret "maas-api-e2e-derive-02ec029f-serving-cert" not found
Sep 06 10:43:09.872307 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:09.872016    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls podName:5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613 nodeName:}" failed. No retries permitted until 2026-09-06 10:43:10.371992606 +0000 UTC m=+773.541842456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls") pod "maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" (UID: "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613") : secret "maas-api-e2e-derive-02ec029f-serving-cert" not found
Sep 06 10:43:09.872573 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.872555    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.874515 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.874492    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:09.879641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:09.879619    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-db8p9\" (UniqueName: \"kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:10.376424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.376377    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:10.378992 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.378971    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") pod \"maas-api-e2e-derive-02ec029f-9b667657d-d5fnh\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:10.413770 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.413737    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:10.417490 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.417465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.419785 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.419764    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-02ec029f-dockercfg-lf6d4\""
Sep 06 10:43:10.419905 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.419779    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-02ec029f\""
Sep 06 10:43:10.428985 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.428960    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:10.477503 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.477468    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.477669 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.477520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz6w8\" (UniqueName: \"kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.569734 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.569697    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:10.579253 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.579217    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.579439 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.579311    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz6w8\" (UniqueName: \"kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.580023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.579983    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.589209 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.589139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz6w8\" (UniqueName: \"kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8\") pod \"payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") " pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.730188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.730126    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:10.847137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.847110    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:10.849241 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:43:10.849214    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae66cba_8b26_4d23_a4d1_d2ccfb4bd613.slice/crio-bd38c2f988493cc09a1d90d9b82acff052fd691daacb8daabc9a0b905760c7d8 WatchSource:0}: Error finding container bd38c2f988493cc09a1d90d9b82acff052fd691daacb8daabc9a0b905760c7d8: Status 404 returned error can't find the container with id bd38c2f988493cc09a1d90d9b82acff052fd691daacb8daabc9a0b905760c7d8
Sep 06 10:43:10.950804 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:10.950766    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" event={"ID":"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613","Type":"ContainerStarted","Data":"bd38c2f988493cc09a1d90d9b82acff052fd691daacb8daabc9a0b905760c7d8"}
Sep 06 10:43:11.004977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.004950    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:11.006846 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:43:11.006820    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca55ec6_e4bb_42d3_af61_29aca18c926a.slice/crio-cd61edc928fa7ba91a02e6f31f970aa620a718a2a0b89343cd79c5749c068c67 WatchSource:0}: Error finding container cd61edc928fa7ba91a02e6f31f970aa620a718a2a0b89343cd79c5749c068c67: Status 404 returned error can't find the container with id cd61edc928fa7ba91a02e6f31f970aa620a718a2a0b89343cd79c5749c068c67
Sep 06 10:43:11.957258 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.957217    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" event={"ID":"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613","Type":"ContainerStarted","Data":"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"}
Sep 06 10:43:11.957723 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.957288    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:11.958722 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.958691    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" event={"ID":"0ca55ec6-e4bb-42d3-af61-29aca18c926a","Type":"ContainerStarted","Data":"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"}
Sep 06 10:43:11.958843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.958735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" event={"ID":"0ca55ec6-e4bb-42d3-af61-29aca18c926a","Type":"ContainerStarted","Data":"cd61edc928fa7ba91a02e6f31f970aa620a718a2a0b89343cd79c5749c068c67"}
Sep 06 10:43:11.958843 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.958756    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:11.972022 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.971969    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" podStartSLOduration=2.576229991 podStartE2EDuration="2.971950685s" totalImagesPullingTime="395.720694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:11.971850459 +0000 UTC m=+775.141700343" watchObservedRunningTime="2026-09-06 10:43:11.971950685 +0000 UTC m=+775.141800559"
Sep 06 10:43:11.987049 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:11.986999    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" podStartSLOduration=1.9869833479999999 podStartE2EDuration="1.986983348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:11.985666247 +0000 UTC m=+775.155516118" watchObservedRunningTime="2026-09-06 10:43:11.986983348 +0000 UTC m=+775.156833219"
Sep 06 10:43:17.968430 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:17.968400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:22.966037 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:22.966005    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:31.150809 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:31.150777    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:43:36.650340 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:36.650304    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:43:37.474993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:37.474960    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:43:38.090047 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.089999    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" podUID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" containerName="maas-api" containerID="cri-o://bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a" gracePeriod=30
Sep 06 10:43:38.340935 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.340866    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:43:38.457006 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.456974    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46k9b\" (UniqueName: \"kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b\") pod \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") "
Sep 06 10:43:38.457288 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.457063    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls\") pod \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") "
Sep 06 10:43:38.457288 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.457113    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca\") pod \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") "
Sep 06 10:43:38.457288 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.457223    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs\") pod \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\" (UID: \"2c81057e-1ae4-423a-9739-f6cb28d2bc8a\") "
Sep 06 10:43:38.457609 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.457452    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2c81057e-1ae4-423a-9739-f6cb28d2bc8a" (UID: "2c81057e-1ae4-423a-9739-f6cb28d2bc8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:43:38.457609 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.457555    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:38.459590 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.459557    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2c81057e-1ae4-423a-9739-f6cb28d2bc8a" (UID: "2c81057e-1ae4-423a-9739-f6cb28d2bc8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:38.459957 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.459935    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2c81057e-1ae4-423a-9739-f6cb28d2bc8a" (UID: "2c81057e-1ae4-423a-9739-f6cb28d2bc8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:38.460066 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.459936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b" (OuterVolumeSpecName: "kube-api-access-46k9b") pod "2c81057e-1ae4-423a-9739-f6cb28d2bc8a" (UID: "2c81057e-1ae4-423a-9739-f6cb28d2bc8a"). InnerVolumeSpecName "kube-api-access-46k9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:38.558703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.558655    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:38.558703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.558685    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:38.558703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:38.558695    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46k9b\" (UniqueName: \"kubernetes.io/projected/2c81057e-1ae4-423a-9739-f6cb28d2bc8a-kube-api-access-46k9b\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:39.097538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.097501    2573 generic.go:356] "Generic (PLEG): container finished" podID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" containerID="bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a" exitCode=0
Sep 06 10:43:39.097977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.097551    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"
Sep 06 10:43:39.097977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.097588    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" event={"ID":"2c81057e-1ae4-423a-9739-f6cb28d2bc8a","Type":"ContainerDied","Data":"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"}
Sep 06 10:43:39.097977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.097649    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2" event={"ID":"2c81057e-1ae4-423a-9739-f6cb28d2bc8a","Type":"ContainerDied","Data":"b4b359cd083e3e72f16db42ff4ce0ab44428987772f14704a8efb0d0417bf9eb"}
Sep 06 10:43:39.097977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.097685    2573 scope.go:117] "RemoveContainer" containerID="bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"
Sep 06 10:43:39.107558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.107542    2573 scope.go:117] "RemoveContainer" containerID="bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"
Sep 06 10:43:39.107856 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:39.107838    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a\": container with ID starting with bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a not found: ID does not exist" containerID="bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"
Sep 06 10:43:39.107920 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.107877    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a"} err="failed to get container status \"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a\": rpc error: code = NotFound desc = could not find container \"bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a\": container with ID starting with bfbc4f51792ad2246487829dc31dc94b6efb81c1af2a2a88938e6d2d838bfb0a not found: ID does not exist"
Sep 06 10:43:39.126311 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.126281    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:43:39.132422 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.132403    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-55111c-58b49cfb45-dhjz2"]
Sep 06 10:43:39.356351 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:39.356254    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" path="/var/lib/kubelet/pods/2c81057e-1ae4-423a-9739-f6cb28d2bc8a/volumes"
Sep 06 10:43:40.490255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.490218    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:40.490709 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.490486    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" podUID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" containerName="maas-api" containerID="cri-o://afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7" gracePeriod=30
Sep 06 10:43:40.547690 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.547657    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:40.547932 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.547886    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" podUID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" containerName="payload-processing" containerID="cri-o://1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260" gracePeriod=30
Sep 06 10:43:40.809363 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.809337    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:40.841407 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.841379    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:40.885950 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.885912    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume\") pod \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") "
Sep 06 10:43:40.886138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.885989    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz6w8\" (UniqueName: \"kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8\") pod \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\" (UID: \"0ca55ec6-e4bb-42d3-af61-29aca18c926a\") "
Sep 06 10:43:40.886138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886067    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs\") pod \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") "
Sep 06 10:43:40.886138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886094    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-db8p9\" (UniqueName: \"kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9\") pod \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") "
Sep 06 10:43:40.886138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886134    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca\") pod \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") "
Sep 06 10:43:40.886360 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886192    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") pod \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\" (UID: \"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613\") "
Sep 06 10:43:40.886458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886344    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ca55ec6-e4bb-42d3-af61-29aca18c926a" (UID: "0ca55ec6-e4bb-42d3-af61-29aca18c926a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:43:40.886547 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886527    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ca55ec6-e4bb-42d3-af61-29aca18c926a-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:40.886608 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.886536    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" (UID: "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:43:40.888610 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.888575    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" (UID: "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:40.888723 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.888677    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9" (OuterVolumeSpecName: "kube-api-access-db8p9") pod "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" (UID: "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613"). InnerVolumeSpecName "kube-api-access-db8p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:40.888723 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.888706    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8" (OuterVolumeSpecName: "kube-api-access-gz6w8") pod "0ca55ec6-e4bb-42d3-af61-29aca18c926a" (UID: "0ca55ec6-e4bb-42d3-af61-29aca18c926a"). InnerVolumeSpecName "kube-api-access-gz6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:40.888799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.888779    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" (UID: "5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:40.987519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.987483    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:40.987519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.987512    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db8p9\" (UniqueName: \"kubernetes.io/projected/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-kube-api-access-db8p9\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:40.987519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.987521    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:40.987519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.987529    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:40.987789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:40.987539    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz6w8\" (UniqueName: \"kubernetes.io/projected/0ca55ec6-e4bb-42d3-af61-29aca18c926a-kube-api-access-gz6w8\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:41.109671 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.109580    2573 generic.go:356] "Generic (PLEG): container finished" podID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" containerID="afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7" exitCode=0
Sep 06 10:43:41.109671 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.109623    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"
Sep 06 10:43:41.109870 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.109675    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" event={"ID":"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613","Type":"ContainerDied","Data":"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"}
Sep 06 10:43:41.109870 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.109747    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh" event={"ID":"5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613","Type":"ContainerDied","Data":"bd38c2f988493cc09a1d90d9b82acff052fd691daacb8daabc9a0b905760c7d8"}
Sep 06 10:43:41.109870 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.109782    2573 scope.go:117] "RemoveContainer" containerID="afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"
Sep 06 10:43:41.111070 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.111050    2573 generic.go:356] "Generic (PLEG): container finished" podID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" containerID="1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260" exitCode=0
Sep 06 10:43:41.111155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.111135    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"
Sep 06 10:43:41.111233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.111157    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" event={"ID":"0ca55ec6-e4bb-42d3-af61-29aca18c926a","Type":"ContainerDied","Data":"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"}
Sep 06 10:43:41.111233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.111218    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd" event={"ID":"0ca55ec6-e4bb-42d3-af61-29aca18c926a","Type":"ContainerDied","Data":"cd61edc928fa7ba91a02e6f31f970aa620a718a2a0b89343cd79c5749c068c67"}
Sep 06 10:43:41.120107 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.120092    2573 scope.go:117] "RemoveContainer" containerID="afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"
Sep 06 10:43:41.120518 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:41.120497    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7\": container with ID starting with afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7 not found: ID does not exist" containerID="afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"
Sep 06 10:43:41.120608 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.120526    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7"} err="failed to get container status \"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7\": rpc error: code = NotFound desc = could not find container \"afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7\": container with ID starting with afcf7bd14416f8b26d286da7609edf980d7ddda1eb505ff1c925c9fd4c5abcf7 not found: ID does not exist"
Sep 06 10:43:41.120608 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.120545    2573 scope.go:117] "RemoveContainer" containerID="1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"
Sep 06 10:43:41.128479 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.128461    2573 scope.go:117] "RemoveContainer" containerID="1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"
Sep 06 10:43:41.129065 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:41.128764    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260\": container with ID starting with 1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260 not found: ID does not exist" containerID="1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"
Sep 06 10:43:41.129065 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.128980    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260"} err="failed to get container status \"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260\": rpc error: code = NotFound desc = could not find container \"1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260\": container with ID starting with 1e628b78d52481dcb4ebf540168a18ed5435a065db5c83a44c5f3f6e2d262260 not found: ID does not exist"
Sep 06 10:43:41.130859 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.130840    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:41.134023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.134003    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-02ec029f-9b667657d-d5fnh"]
Sep 06 10:43:41.142902 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.142877    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:41.146698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.146675    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-02ec029f-7c98987685-qjlxd"]
Sep 06 10:43:41.352717 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.352687    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" path="/var/lib/kubelet/pods/0ca55ec6-e4bb-42d3-af61-29aca18c926a/volumes"
Sep 06 10:43:41.353090 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:41.353077    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" path="/var/lib/kubelet/pods/5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613/volumes"
Sep 06 10:43:46.173639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.173596    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" podUID="de05c560-55a9-4ce4-895a-ebd40885de33" containerName="main" containerID="cri-o://d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873" gracePeriod=45
Sep 06 10:43:46.438316 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.438249    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:46.542848 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.542816    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.542880    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.542946    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xdd6\" (UniqueName: \"kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.542979    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543020    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543265 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543050    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm\") pod \"de05c560-55a9-4ce4-895a-ebd40885de33\" (UID: \"de05c560-55a9-4ce4-895a-ebd40885de33\") "
Sep 06 10:43:46.543265 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543094    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:46.543392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543375    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:46.543552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543525    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache" (OuterVolumeSpecName: "model-cache") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:46.543552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.543536    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home" (OuterVolumeSpecName: "home") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:46.545385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.545359    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:46.545385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.545381    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6" (OuterVolumeSpecName: "kube-api-access-9xdd6") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "kube-api-access-9xdd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:46.545932 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.545914    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm" (OuterVolumeSpecName: "dshm") pod "de05c560-55a9-4ce4-895a-ebd40885de33" (UID: "de05c560-55a9-4ce4-895a-ebd40885de33"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:46.644059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.644029    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:46.644059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.644053    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:46.644059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.644063    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de05c560-55a9-4ce4-895a-ebd40885de33-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:46.644332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.644071    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xdd6\" (UniqueName: \"kubernetes.io/projected/de05c560-55a9-4ce4-895a-ebd40885de33-kube-api-access-9xdd6\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:46.644332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:46.644080    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de05c560-55a9-4ce4-895a-ebd40885de33-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:47.142441 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.142409    2573 generic.go:356] "Generic (PLEG): container finished" podID="de05c560-55a9-4ce4-895a-ebd40885de33" containerID="d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873" exitCode=0
Sep 06 10:43:47.142642 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.142462    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"
Sep 06 10:43:47.142642 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.142509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" event={"ID":"de05c560-55a9-4ce4-895a-ebd40885de33","Type":"ContainerDied","Data":"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"}
Sep 06 10:43:47.142642 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.142563    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489" event={"ID":"de05c560-55a9-4ce4-895a-ebd40885de33","Type":"ContainerDied","Data":"a770b92dedf553046d088f6e347124329d6d7b58a0b6e8f95dfbd81d2d60db63"}
Sep 06 10:43:47.142642 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.142590    2573 scope.go:117] "RemoveContainer" containerID="d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"
Sep 06 10:43:47.151988 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.151971    2573 scope.go:117] "RemoveContainer" containerID="d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"
Sep 06 10:43:47.152303 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:47.152284    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873\": container with ID starting with d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873 not found: ID does not exist" containerID="d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"
Sep 06 10:43:47.152377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.152316    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873"} err="failed to get container status \"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873\": rpc error: code = NotFound desc = could not find container \"d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873\": container with ID starting with d6f096988ddcc07b258a5d59464357a767df5624cc47731c220208dce6af3873 not found: ID does not exist"
Sep 06 10:43:47.161746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.161725    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:43:47.164740 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.164721    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-55111c/autores-model-55111c-kserve-5d57b789d7-82489"]
Sep 06 10:43:47.353289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:47.353259    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de05c560-55a9-4ce4-895a-ebd40885de33" path="/var/lib/kubelet/pods/de05c560-55a9-4ce4-895a-ebd40885de33/volumes"
Sep 06 10:43:51.673887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.673841    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" podUID="4b84f684-020e-45b5-a2ba-36f243efc180" containerName="main" containerID="cri-o://51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563" gracePeriod=45
Sep 06 10:43:51.927806 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.927743    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:43:51.993015 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.992977    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993053    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993076    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993179    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt49f\" (UniqueName: \"kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993225    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir\") pod \"4b84f684-020e-45b5-a2ba-36f243efc180\" (UID: \"4b84f684-020e-45b5-a2ba-36f243efc180\") "
Sep 06 10:43:51.993497 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993377    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home" (OuterVolumeSpecName: "home") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:51.993497 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993428    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache" (OuterVolumeSpecName: "model-cache") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:51.993615 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993556    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:51.993665 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993610    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:51.993665 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.993630    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:51.995435 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.995414    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:43:51.995513 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.995472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f" (OuterVolumeSpecName: "kube-api-access-mt49f") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "kube-api-access-mt49f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:43:51.995513 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:51.995488    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm" (OuterVolumeSpecName: "dshm") pod "4b84f684-020e-45b5-a2ba-36f243efc180" (UID: "4b84f684-020e-45b5-a2ba-36f243efc180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:43:52.094953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.094915    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b84f684-020e-45b5-a2ba-36f243efc180-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:52.094953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.094944    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:52.094953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.094953    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt49f\" (UniqueName: \"kubernetes.io/projected/4b84f684-020e-45b5-a2ba-36f243efc180-kube-api-access-mt49f\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:52.095250 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.094980    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b84f684-020e-45b5-a2ba-36f243efc180-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:43:52.171109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.169019    2573 generic.go:356] "Generic (PLEG): container finished" podID="4b84f684-020e-45b5-a2ba-36f243efc180" containerID="51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563" exitCode=0
Sep 06 10:43:52.171109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.169115    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" event={"ID":"4b84f684-020e-45b5-a2ba-36f243efc180","Type":"ContainerDied","Data":"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"}
Sep 06 10:43:52.171109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.169150    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24" event={"ID":"4b84f684-020e-45b5-a2ba-36f243efc180","Type":"ContainerDied","Data":"13da7be5a199a26cbc4d3621afda9086ce519e5acc12959e065019ba4b4bb741"}
Sep 06 10:43:52.171109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.169191    2573 scope.go:117] "RemoveContainer" containerID="51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"
Sep 06 10:43:52.171109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.169339    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"
Sep 06 10:43:52.179982 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.179961    2573 scope.go:117] "RemoveContainer" containerID="51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"
Sep 06 10:43:52.180281 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:43:52.180260    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563\": container with ID starting with 51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563 not found: ID does not exist" containerID="51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"
Sep 06 10:43:52.180384 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.180288    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563"} err="failed to get container status \"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563\": rpc error: code = NotFound desc = could not find container \"51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563\": container with ID starting with 51ee9b5d0ec67984fc86108aad6b73f814892cd5b11a0f1b1edf2904d41c5563 not found: ID does not exist"
Sep 06 10:43:52.189734 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.189706    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:43:52.193364 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:52.193341    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-511895d8-kserve-86644f7c46-5mq24"]
Sep 06 10:43:53.352764 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:53.352730    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b84f684-020e-45b5-a2ba-36f243efc180" path="/var/lib/kubelet/pods/4b84f684-020e-45b5-a2ba-36f243efc180/volumes"
Sep 06 10:43:54.517242 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517210    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517598    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de05c560-55a9-4ce4-895a-ebd40885de33" containerName="main"
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517610    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="de05c560-55a9-4ce4-895a-ebd40885de33" containerName="main"
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517618    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" containerName="maas-api"
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517625    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" containerName="maas-api"
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517633    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" containerName="payload-processing"
Sep 06 10:43:54.517639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517639    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" containerName="payload-processing"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517647    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" containerName="maas-api"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517654    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" containerName="maas-api"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517671    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b84f684-020e-45b5-a2ba-36f243efc180" containerName="main"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517678    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b84f684-020e-45b5-a2ba-36f243efc180" containerName="main"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517763    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ae66cba-8b26-4d23-a4d1-d2ccfb4bd613" containerName="maas-api"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517774    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="de05c560-55a9-4ce4-895a-ebd40885de33" containerName="main"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517783    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b84f684-020e-45b5-a2ba-36f243efc180" containerName="main"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517790    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c81057e-1ae4-423a-9739-f6cb28d2bc8a" containerName="maas-api"
Sep 06 10:43:54.517864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.517798    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca55ec6-e4bb-42d3-af61-29aca18c926a" containerName="payload-processing"
Sep 06 10:43:54.522346 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.522327    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.524084 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.524067    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-ccf833-serving-cert\""
Sep 06 10:43:54.532443 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.532413    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:43:54.620484 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.620452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.620641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.620498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f486\" (UniqueName: \"kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.620641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.620528    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.620641 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.620608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.721377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.721344    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.721552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.721387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.721552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.721517    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f486\" (UniqueName: \"kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.721638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.721574    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.722220 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.722198    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.723894 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.723879    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.723954 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.723929    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.728307 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.728284    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f486\" (UniqueName: \"kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486\") pod \"maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:54.834964 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:54.834868    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:55.108130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.108046    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:43:55.113999 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:43:55.113967    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a167d80_aee4_49b6_8290_58c5316b6ccb.slice/crio-002a884b068201abe8f8c04d67adaf9655388e0e714c3c9629562f4fb88c7005 WatchSource:0}: Error finding container 002a884b068201abe8f8c04d67adaf9655388e0e714c3c9629562f4fb88c7005: Status 404 returned error can't find the container with id 002a884b068201abe8f8c04d67adaf9655388e0e714c3c9629562f4fb88c7005
Sep 06 10:43:55.120262 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.120238    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:43:55.125099 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.125079    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.127019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.126998    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ccf833\""
Sep 06 10:43:55.127110 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.127077    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ccf833-dockercfg-s5zlc\""
Sep 06 10:43:55.134372 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.134349    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:43:55.186882 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.186848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" event={"ID":"7a167d80-aee4-49b6-8290-58c5316b6ccb","Type":"ContainerStarted","Data":"002a884b068201abe8f8c04d67adaf9655388e0e714c3c9629562f4fb88c7005"}
Sep 06 10:43:55.226569 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.226536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.226744 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.226620    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpq7\" (UniqueName: \"kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.327739 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.327693    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.327941 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.327790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpq7\" (UniqueName: \"kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.328454 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.328430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.335437 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.335414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpq7\" (UniqueName: \"kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7\") pod \"payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.437307 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.437279    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:55.715382 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:55.715301    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:43:55.717667 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:43:55.717641    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbc7119_9f3c_4abf_b2b7_208e963ac23b.slice/crio-efcdf7264c38a2e1c4e357c82df465cb432cf6003f278c1266f18058799b5892 WatchSource:0}: Error finding container efcdf7264c38a2e1c4e357c82df465cb432cf6003f278c1266f18058799b5892: Status 404 returned error can't find the container with id efcdf7264c38a2e1c4e357c82df465cb432cf6003f278c1266f18058799b5892
Sep 06 10:43:56.195056 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.195021    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" event={"ID":"7a167d80-aee4-49b6-8290-58c5316b6ccb","Type":"ContainerStarted","Data":"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"}
Sep 06 10:43:56.195255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.195185    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:43:56.196809 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.196779    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" event={"ID":"3dbc7119-9f3c-4abf-b2b7-208e963ac23b","Type":"ContainerStarted","Data":"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"}
Sep 06 10:43:56.196957 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.196823    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" event={"ID":"3dbc7119-9f3c-4abf-b2b7-208e963ac23b","Type":"ContainerStarted","Data":"efcdf7264c38a2e1c4e357c82df465cb432cf6003f278c1266f18058799b5892"}
Sep 06 10:43:56.196957 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.196879    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:43:56.212149 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.212108    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" podStartSLOduration=1.823920803 podStartE2EDuration="2.212095199s" totalImagesPullingTime="388.174396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:56.210898314 +0000 UTC m=+819.380748210" watchObservedRunningTime="2026-09-06 10:43:56.212095199 +0000 UTC m=+819.381945108"
Sep 06 10:43:56.228347 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:43:56.228304    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" podStartSLOduration=1.228290514 podStartE2EDuration="1.228290514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:43:56.227471523 +0000 UTC m=+819.397321395" watchObservedRunningTime="2026-09-06 10:43:56.228290514 +0000 UTC m=+819.398140385"
Sep 06 10:44:02.207453 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:02.207425    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:44:06.307405 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.307371    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:44:06.311050 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.311032    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.313211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.313188    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-636b04\"/\"kube-root-ca.crt\""
Sep 06 10:44:06.314039 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.314007    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-636b04\"/\"default-dockercfg-tffcg\""
Sep 06 10:44:06.314293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.314042    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-636b04\"/\"openshift-service-ca.crt\""
Sep 06 10:44:06.314293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.314009    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-636b04\"/\"test-model-636b04-kserve-self-signed-certs\""
Sep 06 10:44:06.317799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.317749    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:44:06.336317 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336286    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.336459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336354    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.336602 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336582    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.336800 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336634    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.336892 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336877    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.336930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.336908    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwq62\" (UniqueName: \"kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438036 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.437993    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438081    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438121    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwq62\" (UniqueName: \"kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438198    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438512 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438301    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438512 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438478    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438625 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438559    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.438684 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.438631    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.440562 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.440534    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.441003 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.440980    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.445544 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.445515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwq62\" (UniqueName: \"kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62\") pod \"test-model-636b04-kserve-bc755745f-pgd2p\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") " pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.623833 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.623758    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:06.896615 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:06.896589    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:44:06.898672 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:44:06.898601    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecf32cb3_9bc9_4917_a0d3_d223723c910a.slice/crio-f07e2e4fa9733df5d8fca31a1fc09d1c33dd26e54d7263618edbc58e88f44a75 WatchSource:0}: Error finding container f07e2e4fa9733df5d8fca31a1fc09d1c33dd26e54d7263618edbc58e88f44a75: Status 404 returned error can't find the container with id f07e2e4fa9733df5d8fca31a1fc09d1c33dd26e54d7263618edbc58e88f44a75
Sep 06 10:44:07.204312 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:07.204225    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:44:07.254244 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:07.254210    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" event={"ID":"ecf32cb3-9bc9-4917-a0d3-d223723c910a","Type":"ContainerStarted","Data":"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"}
Sep 06 10:44:07.254412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:07.254253    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" event={"ID":"ecf32cb3-9bc9-4917-a0d3-d223723c910a","Type":"ContainerStarted","Data":"f07e2e4fa9733df5d8fca31a1fc09d1c33dd26e54d7263618edbc58e88f44a75"}
Sep 06 10:44:07.273339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:07.273267    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" podStartSLOduration=1.27324488 podStartE2EDuration="1.27324488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:44:07.272713505 +0000 UTC m=+830.442563390" watchObservedRunningTime="2026-09-06 10:44:07.27324488 +0000 UTC m=+830.443094754"
Sep 06 10:44:16.624298 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:16.624257    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:16.624791 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:16.624310    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:16.636694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:16.636668    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:17.315486 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:17.315457    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:44:27.326175 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.326131    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:27.330953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.330929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.333222 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.333199    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ccf833\"/\"openshift-service-ca.crt\""
Sep 06 10:44:27.333339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.333204    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ccf833\"/\"kube-root-ca.crt\""
Sep 06 10:44:27.334146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.334128    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ccf833\"/\"explicit-model-ccf833-kserve-self-signed-certs\""
Sep 06 10:44:27.334258 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.334207    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ccf833\"/\"default-dockercfg-l6qm5\""
Sep 06 10:44:27.338221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.338196    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:27.436963 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.436930    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56gcc\" (UniqueName: \"kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.437143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.436988    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.437143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.437012    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.437143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.437028    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.437325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.437157    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.437325 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.437270    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538237 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538422 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56gcc\" (UniqueName: \"kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538422 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538381    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538551 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538420    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538551 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538551 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538699 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538598    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538765    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.538916 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.538898    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.540649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.540632    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.540893 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.540874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.546538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.546515    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56gcc\" (UniqueName: \"kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc\") pod \"explicit-model-ccf833-kserve-dbcc666fc-64bwl\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") " pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.640864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.640784    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:27.914367 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:27.914343    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:27.916749 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:44:27.916723    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaac942f_1234_4f00_847d_dd3f941e1589.slice/crio-01073f495ead3985fb10e5707f6f6f5484168750b52c41e55cb74252bdce3a61 WatchSource:0}: Error finding container 01073f495ead3985fb10e5707f6f6f5484168750b52c41e55cb74252bdce3a61: Status 404 returned error can't find the container with id 01073f495ead3985fb10e5707f6f6f5484168750b52c41e55cb74252bdce3a61
Sep 06 10:44:28.358938 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:28.358848    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" event={"ID":"baac942f-1234-4f00-847d-dd3f941e1589","Type":"ContainerStarted","Data":"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"}
Sep 06 10:44:28.358938 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:28.358900    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" event={"ID":"baac942f-1234-4f00-847d-dd3f941e1589","Type":"ContainerStarted","Data":"01073f495ead3985fb10e5707f6f6f5484168750b52c41e55cb74252bdce3a61"}
Sep 06 10:44:28.375480 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:28.375427    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" podStartSLOduration=1.375411338 podStartE2EDuration="1.375411338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:44:28.373581037 +0000 UTC m=+851.543430908" watchObservedRunningTime="2026-09-06 10:44:28.375411338 +0000 UTC m=+851.545261210"
Sep 06 10:44:29.954377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:29.954339    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:37.641813 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:37.641779    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:40.525155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.523914    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:44:40.525155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.524217    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" podUID="7a167d80-aee4-49b6-8290-58c5316b6ccb" containerName="maas-api" containerID="cri-o://62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820" gracePeriod=30
Sep 06 10:44:40.579315 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.579279    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:44:40.579538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.579508    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" podUID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" containerName="payload-pre-processing" containerID="cri-o://1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7" gracePeriod=30
Sep 06 10:44:40.835666 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.835646    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:44:40.866612 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.866579    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:44:40.974471 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974434    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca\") pod \"7a167d80-aee4-49b6-8290-58c5316b6ccb\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") "
Sep 06 10:44:40.974670 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974480    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls\") pod \"7a167d80-aee4-49b6-8290-58c5316b6ccb\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") "
Sep 06 10:44:40.974670 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974499    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume\") pod \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") "
Sep 06 10:44:40.974670 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974550    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f486\" (UniqueName: \"kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486\") pod \"7a167d80-aee4-49b6-8290-58c5316b6ccb\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") "
Sep 06 10:44:40.974670 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974614    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs\") pod \"7a167d80-aee4-49b6-8290-58c5316b6ccb\" (UID: \"7a167d80-aee4-49b6-8290-58c5316b6ccb\") "
Sep 06 10:44:40.974905 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974696    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smpq7\" (UniqueName: \"kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7\") pod \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\" (UID: \"3dbc7119-9f3c-4abf-b2b7-208e963ac23b\") "
Sep 06 10:44:40.974905 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974849    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a167d80-aee4-49b6-8290-58c5316b6ccb" (UID: "7a167d80-aee4-49b6-8290-58c5316b6ccb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:40.974999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.974922    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3dbc7119-9f3c-4abf-b2b7-208e963ac23b" (UID: "3dbc7119-9f3c-4abf-b2b7-208e963ac23b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:44:40.975044 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.975028    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a167d80-aee4-49b6-8290-58c5316b6ccb-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:40.975093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.975041    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:40.976816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.976782    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a167d80-aee4-49b6-8290-58c5316b6ccb" (UID: "7a167d80-aee4-49b6-8290-58c5316b6ccb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:40.976998 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.976921    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a167d80-aee4-49b6-8290-58c5316b6ccb" (UID: "7a167d80-aee4-49b6-8290-58c5316b6ccb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:40.976998 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.976975    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486" (OuterVolumeSpecName: "kube-api-access-6f486") pod "7a167d80-aee4-49b6-8290-58c5316b6ccb" (UID: "7a167d80-aee4-49b6-8290-58c5316b6ccb"). InnerVolumeSpecName "kube-api-access-6f486". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:40.977076 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:40.977021    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7" (OuterVolumeSpecName: "kube-api-access-smpq7") pod "3dbc7119-9f3c-4abf-b2b7-208e963ac23b" (UID: "3dbc7119-9f3c-4abf-b2b7-208e963ac23b"). InnerVolumeSpecName "kube-api-access-smpq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:41.075858 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.075778    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.075858 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.075808    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smpq7\" (UniqueName: \"kubernetes.io/projected/3dbc7119-9f3c-4abf-b2b7-208e963ac23b-kube-api-access-smpq7\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.075858 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.075818    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a167d80-aee4-49b6-8290-58c5316b6ccb-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.075858 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.075826    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f486\" (UniqueName: \"kubernetes.io/projected/7a167d80-aee4-49b6-8290-58c5316b6ccb-kube-api-access-6f486\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:41.420647 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.420613    2573 generic.go:356] "Generic (PLEG): container finished" podID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" containerID="1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7" exitCode=0
Sep 06 10:44:41.420846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.420663    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"
Sep 06 10:44:41.420846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.420707    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" event={"ID":"3dbc7119-9f3c-4abf-b2b7-208e963ac23b","Type":"ContainerDied","Data":"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"}
Sep 06 10:44:41.420846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.420768    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r" event={"ID":"3dbc7119-9f3c-4abf-b2b7-208e963ac23b","Type":"ContainerDied","Data":"efcdf7264c38a2e1c4e357c82df465cb432cf6003f278c1266f18058799b5892"}
Sep 06 10:44:41.420846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.420797    2573 scope.go:117] "RemoveContainer" containerID="1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"
Sep 06 10:44:41.422477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.422285    2573 generic.go:356] "Generic (PLEG): container finished" podID="7a167d80-aee4-49b6-8290-58c5316b6ccb" containerID="62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820" exitCode=0
Sep 06 10:44:41.422477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.422330    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" event={"ID":"7a167d80-aee4-49b6-8290-58c5316b6ccb","Type":"ContainerDied","Data":"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"}
Sep 06 10:44:41.422477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.422350    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"
Sep 06 10:44:41.422477 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.422366    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859" event={"ID":"7a167d80-aee4-49b6-8290-58c5316b6ccb","Type":"ContainerDied","Data":"002a884b068201abe8f8c04d67adaf9655388e0e714c3c9629562f4fb88c7005"}
Sep 06 10:44:41.430345 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.430311    2573 scope.go:117] "RemoveContainer" containerID="1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"
Sep 06 10:44:41.430658 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:44:41.430636    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7\": container with ID starting with 1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7 not found: ID does not exist" containerID="1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"
Sep 06 10:44:41.430763 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.430663    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7"} err="failed to get container status \"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7\": rpc error: code = NotFound desc = could not find container \"1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7\": container with ID starting with 1b4824ea6bbb3908f7788987845a334fad526231b1a3b2c43cdc36e050f86fb7 not found: ID does not exist"
Sep 06 10:44:41.430763 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.430679    2573 scope.go:117] "RemoveContainer" containerID="62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"
Sep 06 10:44:41.438130 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.438109    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:44:41.439469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.439434    2573 scope.go:117] "RemoveContainer" containerID="62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"
Sep 06 10:44:41.439793 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:44:41.439772    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820\": container with ID starting with 62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820 not found: ID does not exist" containerID="62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"
Sep 06 10:44:41.439874 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.439803    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820"} err="failed to get container status \"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820\": rpc error: code = NotFound desc = could not find container \"62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820\": container with ID starting with 62e0f2952cd58a02d2ea2c1b540d9ca23618043c3fa225d5388dd9e9f311d820 not found: ID does not exist"
Sep 06 10:44:41.441350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.441324    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ccf833-7c9c6cf986-zfw9r"]
Sep 06 10:44:41.450176 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.450143    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:44:41.453016 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:41.452996    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-ccf833-65f6f7bf98-tc859"]
Sep 06 10:44:43.354809 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:43.354773    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" path="/var/lib/kubelet/pods/3dbc7119-9f3c-4abf-b2b7-208e963ac23b/volumes"
Sep 06 10:44:43.355188 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:43.355156    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a167d80-aee4-49b6-8290-58c5316b6ccb" path="/var/lib/kubelet/pods/7a167d80-aee4-49b6-8290-58c5316b6ccb/volumes"
Sep 06 10:44:45.390357 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.390312    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" podUID="baac942f-1234-4f00-847d-dd3f941e1589" containerName="main" containerID="cri-o://99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8" gracePeriod=45
Sep 06 10:44:45.640322 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.640294    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:45.718971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.718935    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.718992    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56gcc\" (UniqueName: \"kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719034    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719114    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719136    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719221 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719208    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home\") pod \"baac942f-1234-4f00-847d-dd3f941e1589\" (UID: \"baac942f-1234-4f00-847d-dd3f941e1589\") "
Sep 06 10:44:45.719531 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719443    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:45.719531 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home" (OuterVolumeSpecName: "home") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:45.719651 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719592    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:45.719651 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719611    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:45.719651 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.719615    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache" (OuterVolumeSpecName: "model-cache") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:45.721605 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.721579    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc" (OuterVolumeSpecName: "kube-api-access-56gcc") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "kube-api-access-56gcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:44:45.721605 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.721591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm" (OuterVolumeSpecName: "dshm") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:44:45.721765 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.721684    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "baac942f-1234-4f00-847d-dd3f941e1589" (UID: "baac942f-1234-4f00-847d-dd3f941e1589"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:44:45.820823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.820779    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56gcc\" (UniqueName: \"kubernetes.io/projected/baac942f-1234-4f00-847d-dd3f941e1589-kube-api-access-56gcc\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:45.820823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.820814    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:45.820823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.820823    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baac942f-1234-4f00-847d-dd3f941e1589-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:45.820823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:45.820831    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baac942f-1234-4f00-847d-dd3f941e1589-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:44:46.450711 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.450680    2573 generic.go:356] "Generic (PLEG): container finished" podID="baac942f-1234-4f00-847d-dd3f941e1589" containerID="99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8" exitCode=0
Sep 06 10:44:46.451197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.450720    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"
Sep 06 10:44:46.451197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.450762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" event={"ID":"baac942f-1234-4f00-847d-dd3f941e1589","Type":"ContainerDied","Data":"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"}
Sep 06 10:44:46.451197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.450802    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl" event={"ID":"baac942f-1234-4f00-847d-dd3f941e1589","Type":"ContainerDied","Data":"01073f495ead3985fb10e5707f6f6f5484168750b52c41e55cb74252bdce3a61"}
Sep 06 10:44:46.451197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.450819    2573 scope.go:117] "RemoveContainer" containerID="99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"
Sep 06 10:44:46.460269 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.460253    2573 scope.go:117] "RemoveContainer" containerID="99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"
Sep 06 10:44:46.460568 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:44:46.460546    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8\": container with ID starting with 99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8 not found: ID does not exist" containerID="99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"
Sep 06 10:44:46.460628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.460577    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8"} err="failed to get container status \"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8\": rpc error: code = NotFound desc = could not find container \"99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8\": container with ID starting with 99457165de5b9866fec85c5c3d827348f27066c828fcd360b14bbc3dd0b5c2d8 not found: ID does not exist"
Sep 06 10:44:46.469961 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.469937    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:46.474922 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:46.474900    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ccf833/explicit-model-ccf833-kserve-dbcc666fc-64bwl"]
Sep 06 10:44:47.353691 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:47.353659    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baac942f-1234-4f00-847d-dd3f941e1589" path="/var/lib/kubelet/pods/baac942f-1234-4f00-847d-dd3f941e1589/volumes"
Sep 06 10:44:56.622760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.622722    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:44:56.623300 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623281    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a167d80-aee4-49b6-8290-58c5316b6ccb" containerName="maas-api"
Sep 06 10:44:56.623392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623304    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a167d80-aee4-49b6-8290-58c5316b6ccb" containerName="maas-api"
Sep 06 10:44:56.623392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623315    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baac942f-1234-4f00-847d-dd3f941e1589" containerName="main"
Sep 06 10:44:56.623392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623323    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="baac942f-1234-4f00-847d-dd3f941e1589" containerName="main"
Sep 06 10:44:56.623392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623354    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" containerName="payload-pre-processing"
Sep 06 10:44:56.623392 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623363    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" containerName="payload-pre-processing"
Sep 06 10:44:56.623639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623476    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a167d80-aee4-49b6-8290-58c5316b6ccb" containerName="maas-api"
Sep 06 10:44:56.623639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623492    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3dbc7119-9f3c-4abf-b2b7-208e963ac23b" containerName="payload-pre-processing"
Sep 06 10:44:56.623639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.623502    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="baac942f-1234-4f00-847d-dd3f941e1589" containerName="main"
Sep 06 10:44:56.627969 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.627950    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.629929 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.629910    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-77b0cb\"/\"kube-root-ca.crt\""
Sep 06 10:44:56.630518 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.630497    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-77b0cb\"/\"openshift-service-ca.crt\""
Sep 06 10:44:56.630662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.630505    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-77b0cb\"/\"default-dockercfg-l87lz\""
Sep 06 10:44:56.630662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.630631    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-77b0cb\"/\"noten-model-77b0cb-kserve-self-signed-certs\""
Sep 06 10:44:56.633779 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.633756    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:44:56.725997 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.725955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.726211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.726013    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvgkt\" (UniqueName: \"kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.726211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.726065    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.726211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.726089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.726211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.726138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.726211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.726204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827175 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827130    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827208    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvgkt\" (UniqueName: \"kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827317    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827375 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827359    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827769 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827743    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827856 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827764    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.827856 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.827800    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.829570 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.829549    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.829818 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.829798    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.834036 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.834013    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvgkt\" (UniqueName: \"kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt\") pod \"noten-model-77b0cb-kserve-b78f87f99-l8w2x\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") " pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:56.939581 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:56.939543    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:44:57.210886 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:57.210861    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:44:57.213034 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:44:57.213006    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod824e5b2e_aa3e_4516_9aa7_84b3bec81159.slice/crio-a66f39b3b5718bea03661e2d2eac541b6a17c8c82d840ab326b1f993a8c8df32 WatchSource:0}: Error finding container a66f39b3b5718bea03661e2d2eac541b6a17c8c82d840ab326b1f993a8c8df32: Status 404 returned error can't find the container with id a66f39b3b5718bea03661e2d2eac541b6a17c8c82d840ab326b1f993a8c8df32
Sep 06 10:44:57.511072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:57.510984    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" event={"ID":"824e5b2e-aa3e-4516-9aa7-84b3bec81159","Type":"ContainerStarted","Data":"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"}
Sep 06 10:44:57.511072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:57.511027    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" event={"ID":"824e5b2e-aa3e-4516-9aa7-84b3bec81159","Type":"ContainerStarted","Data":"a66f39b3b5718bea03661e2d2eac541b6a17c8c82d840ab326b1f993a8c8df32"}
Sep 06 10:44:57.525670 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:44:57.525624    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" podStartSLOduration=1.525606624 podStartE2EDuration="1.525606624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:44:57.524568378 +0000 UTC m=+880.694418241" watchObservedRunningTime="2026-09-06 10:44:57.525606624 +0000 UTC m=+880.695456514"
Sep 06 10:45:01.579311 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:01.579262    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:45:06.939875 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:06.939831    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:45:14.309007 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:14.308970    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:45:16.600626 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:16.600575    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" podUID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" containerName="main" containerID="cri-o://273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d" gracePeriod=45
Sep 06 10:45:16.855907 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:16.855846    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:45:17.035755 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035717    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.035755 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035759    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.036023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035815    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.036023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035897    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.036023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035943    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.036023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.035978    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvgkt\" (UniqueName: \"kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt\") pod \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\" (UID: \"824e5b2e-aa3e-4516-9aa7-84b3bec81159\") "
Sep 06 10:45:17.036266 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036149    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home" (OuterVolumeSpecName: "home") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:17.036266 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036220    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:17.036337 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036257    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache" (OuterVolumeSpecName: "model-cache") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:17.036386 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036371    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.036438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036391    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.036438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.036406    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.038248 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.038226    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm" (OuterVolumeSpecName: "dshm") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:17.038348 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.038321    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:17.038348 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.038325    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt" (OuterVolumeSpecName: "kube-api-access-bvgkt") pod "824e5b2e-aa3e-4516-9aa7-84b3bec81159" (UID: "824e5b2e-aa3e-4516-9aa7-84b3bec81159"). InnerVolumeSpecName "kube-api-access-bvgkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:17.137095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.137004    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvgkt\" (UniqueName: \"kubernetes.io/projected/824e5b2e-aa3e-4516-9aa7-84b3bec81159-kube-api-access-bvgkt\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.137095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.137039    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/824e5b2e-aa3e-4516-9aa7-84b3bec81159-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.137095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.137048    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/824e5b2e-aa3e-4516-9aa7-84b3bec81159-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:17.610624 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.610591    2573 generic.go:356] "Generic (PLEG): container finished" podID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" containerID="273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d" exitCode=0
Sep 06 10:45:17.611073 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.610650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"
Sep 06 10:45:17.611073 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.610657    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" event={"ID":"824e5b2e-aa3e-4516-9aa7-84b3bec81159","Type":"ContainerDied","Data":"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"}
Sep 06 10:45:17.611073 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.610689    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x" event={"ID":"824e5b2e-aa3e-4516-9aa7-84b3bec81159","Type":"ContainerDied","Data":"a66f39b3b5718bea03661e2d2eac541b6a17c8c82d840ab326b1f993a8c8df32"}
Sep 06 10:45:17.611073 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.610705    2573 scope.go:117] "RemoveContainer" containerID="273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"
Sep 06 10:45:17.619107 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.619090    2573 scope.go:117] "RemoveContainer" containerID="273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"
Sep 06 10:45:17.619411 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:17.619388    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d\": container with ID starting with 273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d not found: ID does not exist" containerID="273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"
Sep 06 10:45:17.619471 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.619419    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d"} err="failed to get container status \"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d\": rpc error: code = NotFound desc = could not find container \"273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d\": container with ID starting with 273f9516a6e489ce49a6f1c5854c9ccdf55e21a9fd2701308557c91b5981470d not found: ID does not exist"
Sep 06 10:45:17.626941 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.626917    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:45:17.629090 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:17.629069    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-77b0cb/noten-model-77b0cb-kserve-b78f87f99-l8w2x"]
Sep 06 10:45:19.352635 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:19.352600    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" path="/var/lib/kubelet/pods/824e5b2e-aa3e-4516-9aa7-84b3bec81159/volumes"
Sep 06 10:45:20.960702 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.960664    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:45:20.961291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.961271    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" containerName="main"
Sep 06 10:45:20.961291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.961293    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" containerName="main"
Sep 06 10:45:20.961436 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.961392    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="824e5b2e-aa3e-4516-9aa7-84b3bec81159" containerName="main"
Sep 06 10:45:20.965953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.965933    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:20.968262 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.968230    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lctxr\" (UniqueName: \"kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:20.968385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.968310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:20.968385 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.968341    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2313d276-serving-cert\""
Sep 06 10:45:20.968509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.968416    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:20.968509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.968446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:20.973081 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:20.973060    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:45:21.068959 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.068929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.069153 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.068983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.069153 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.069004    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.069153 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.069036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lctxr\" (UniqueName: \"kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.069153 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:21.069092    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-serving-cert: secret "maas-api-e2e-mt-2313d276-serving-cert" not found
Sep 06 10:45:21.069425 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:21.069180    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls podName:60b8ee5c-aaed-4720-9f98-325b8c75cfad nodeName:}" failed. No retries permitted until 2026-09-06 10:45:21.569142961 +0000 UTC m=+904.738992819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls") pod "maas-api-e2e-mt-2313d276-5f4864898f-d2g57" (UID: "60b8ee5c-aaed-4720-9f98-325b8c75cfad") : secret "maas-api-e2e-mt-2313d276-serving-cert" not found
Sep 06 10:45:21.069693 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.069672    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.071593 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.071572    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.076772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.076751    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lctxr\" (UniqueName: \"kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.573220 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.573148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.576083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.576051    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") pod \"maas-api-e2e-mt-2313d276-5f4864898f-d2g57\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.576951 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.576930    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:45:21.578818 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.578795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:21.580880 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.580862    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.583060 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.582977    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2313d276\""
Sep 06 10:45:21.583359 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.583328    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2313d276-dockercfg-nk2md\""
Sep 06 10:45:21.592654 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.592634    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:45:21.673735 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.673707    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.673895 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.673830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvhb\" (UniqueName: \"kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.774788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.774748    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvhb\" (UniqueName: \"kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.774788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.774797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.775540 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.775520    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.782054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.782028    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvhb\" (UniqueName: \"kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb\") pod \"payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:21.856836 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.856815    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:45:21.859154 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:45:21.859130    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b8ee5c_aaed_4720_9f98_325b8c75cfad.slice/crio-6524eb5b8e6d0288c9746def0f11ef27f431b3def017aa44cce4a6ad82dc541b WatchSource:0}: Error finding container 6524eb5b8e6d0288c9746def0f11ef27f431b3def017aa44cce4a6ad82dc541b: Status 404 returned error can't find the container with id 6524eb5b8e6d0288c9746def0f11ef27f431b3def017aa44cce4a6ad82dc541b
Sep 06 10:45:21.911727 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:21.911688    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:22.182671 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.182573    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:45:22.185409 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:45:22.185380    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c0d11e_9752_4bc3_a814_5e1d205aa74f.slice/crio-47c7c8d091144f251dd12583a8f335a57851f733e4f59a7c704c5d0f247294b9 WatchSource:0}: Error finding container 47c7c8d091144f251dd12583a8f335a57851f733e4f59a7c704c5d0f247294b9: Status 404 returned error can't find the container with id 47c7c8d091144f251dd12583a8f335a57851f733e4f59a7c704c5d0f247294b9
Sep 06 10:45:22.639530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.639498    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" event={"ID":"14c0d11e-9752-4bc3-a814-5e1d205aa74f","Type":"ContainerStarted","Data":"8f6f611aeb868cf13edcbf56562ca68d28792d7f9b0d9232732aa178c8c1ac87"}
Sep 06 10:45:22.639727 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.639547    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" event={"ID":"14c0d11e-9752-4bc3-a814-5e1d205aa74f","Type":"ContainerStarted","Data":"47c7c8d091144f251dd12583a8f335a57851f733e4f59a7c704c5d0f247294b9"}
Sep 06 10:45:22.639727 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.639621    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:22.641172 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.641136    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" event={"ID":"60b8ee5c-aaed-4720-9f98-325b8c75cfad","Type":"ContainerStarted","Data":"fc4ce8acb7da1008c286d6643651c1cf4d57bc10557a3fe6838616d78edff573"}
Sep 06 10:45:22.641274 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.641192    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" event={"ID":"60b8ee5c-aaed-4720-9f98-325b8c75cfad","Type":"ContainerStarted","Data":"6524eb5b8e6d0288c9746def0f11ef27f431b3def017aa44cce4a6ad82dc541b"}
Sep 06 10:45:22.641274 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.641223    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:22.654738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.654697    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" podStartSLOduration=1.6546847599999999 podStartE2EDuration="1.65468476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:22.652094481 +0000 UTC m=+905.821944353" watchObservedRunningTime="2026-09-06 10:45:22.65468476 +0000 UTC m=+905.824534653"
Sep 06 10:45:22.666691 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:22.666648    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" podStartSLOduration=2.074654095 podStartE2EDuration="2.666631468s" totalImagesPullingTime="591.977373ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:22.665589455 +0000 UTC m=+905.835439338" watchObservedRunningTime="2026-09-06 10:45:22.666631468 +0000 UTC m=+905.836481341"
Sep 06 10:45:25.077840 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.077798    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:45:25.078636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.078605    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" podUID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" containerName="maas-api" containerID="cri-o://2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c" gracePeriod=30
Sep 06 10:45:25.543871 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.543841    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:45:25.610126 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.610090    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfj75\" (UniqueName: \"kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75\") pod \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") "
Sep 06 10:45:25.610350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.610140    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") pod \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") "
Sep 06 10:45:25.610350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.610211    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca\") pod \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") "
Sep 06 10:45:25.610350 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.610252    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs\") pod \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\" (UID: \"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1\") "
Sep 06 10:45:25.610608 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.610556    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" (UID: "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:45:25.612445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.612408    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" (UID: "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:25.612556 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.612472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" (UID: "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:25.612556 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.612466    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75" (OuterVolumeSpecName: "kube-api-access-wfj75") pod "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" (UID: "75d26e42-a741-4c6a-8ab6-b2778ae8cdb1"). InnerVolumeSpecName "kube-api-access-wfj75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:25.659481 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.659447    2573 generic.go:356] "Generic (PLEG): container finished" podID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" containerID="2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c" exitCode=0
Sep 06 10:45:25.659616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.659495    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"
Sep 06 10:45:25.659616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.659538    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" event={"ID":"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1","Type":"ContainerDied","Data":"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"}
Sep 06 10:45:25.659616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.659594    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq" event={"ID":"75d26e42-a741-4c6a-8ab6-b2778ae8cdb1","Type":"ContainerDied","Data":"23760c14daddfea27c60603ffe93a29aed37d5b6b922d34ce8516d5b217a5b91"}
Sep 06 10:45:25.659765 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.659618    2573 scope.go:117] "RemoveContainer" containerID="2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"
Sep 06 10:45:25.669212 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.669185    2573 scope.go:117] "RemoveContainer" containerID="2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"
Sep 06 10:45:25.669530 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:25.669503    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c\": container with ID starting with 2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c not found: ID does not exist" containerID="2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"
Sep 06 10:45:25.669610 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.669536    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c"} err="failed to get container status \"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c\": rpc error: code = NotFound desc = could not find container \"2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c\": container with ID starting with 2dc299bdd8565569da7b67fa66fa6474e7815432920dc0442021f4c55854044c not found: ID does not exist"
Sep 06 10:45:25.682155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.682129    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:45:25.685852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.685830    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-232aeb-7798d85b64-6sjzq"]
Sep 06 10:45:25.711199 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.711151    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfj75\" (UniqueName: \"kubernetes.io/projected/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-kube-api-access-wfj75\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:25.711199 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.711200    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:25.711333 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.711211    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:25.711333 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:25.711219    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:27.353026 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:27.352988    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" path="/var/lib/kubelet/pods/75d26e42-a741-4c6a-8ab6-b2778ae8cdb1/volumes"
Sep 06 10:45:28.654800 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:28.654772    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:45:29.329706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.329666    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" podUID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" containerName="main" containerID="cri-o://d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7" gracePeriod=45
Sep 06 10:45:29.584540 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.584482    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:45:29.647889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.647857    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.647912    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.647948    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.647995    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648018    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwq62\" (UniqueName: \"kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648313 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648074    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm\") pod \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\" (UID: \"ecf32cb3-9bc9-4917-a0d3-d223723c910a\") "
Sep 06 10:45:29.648313 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648152    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:29.648313 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648210    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home" (OuterVolumeSpecName: "home") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:29.648473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648321    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:29.648473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648388    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:29.648473 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.648401    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:29.650470 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.650426    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm" (OuterVolumeSpecName: "dshm") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:45:29.650470 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.650437    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62" (OuterVolumeSpecName: "kube-api-access-dwq62") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "kube-api-access-dwq62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:29.650620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.650476    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ecf32cb3-9bc9-4917-a0d3-d223723c910a" (UID: "ecf32cb3-9bc9-4917-a0d3-d223723c910a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:29.689233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.689200    2573 generic.go:356] "Generic (PLEG): container finished" podID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" containerID="d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7" exitCode=0
Sep 06 10:45:29.689582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.689292    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"
Sep 06 10:45:29.689582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.689327    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" event={"ID":"ecf32cb3-9bc9-4917-a0d3-d223723c910a","Type":"ContainerDied","Data":"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"}
Sep 06 10:45:29.689582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.689381    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p" event={"ID":"ecf32cb3-9bc9-4917-a0d3-d223723c910a","Type":"ContainerDied","Data":"f07e2e4fa9733df5d8fca31a1fc09d1c33dd26e54d7263618edbc58e88f44a75"}
Sep 06 10:45:29.689582 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.689410    2573 scope.go:117] "RemoveContainer" containerID="d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"
Sep 06 10:45:29.698960 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.698939    2573 scope.go:117] "RemoveContainer" containerID="d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"
Sep 06 10:45:29.699291 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:29.699272    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7\": container with ID starting with d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7 not found: ID does not exist" containerID="d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"
Sep 06 10:45:29.699404 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.699300    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7"} err="failed to get container status \"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7\": rpc error: code = NotFound desc = could not find container \"d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7\": container with ID starting with d547a941a41b379dc481f0f2f68ce1f876fe5d9dc47a841e65f08697232b9ff7 not found: ID does not exist"
Sep 06 10:45:29.708471 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.708422    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:45:29.710311 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.710290    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-636b04/test-model-636b04-kserve-bc755745f-pgd2p"]
Sep 06 10:45:29.749530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.749496    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:29.749530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.749525    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:29.749530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.749533    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwq62\" (UniqueName: \"kubernetes.io/projected/ecf32cb3-9bc9-4917-a0d3-d223723c910a-kube-api-access-dwq62\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:29.749751 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:29.749542    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ecf32cb3-9bc9-4917-a0d3-d223723c910a-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:31.352389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:31.352356    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" path="/var/lib/kubelet/pods/ecf32cb3-9bc9-4917-a0d3-d223723c910a/volumes"
Sep 06 10:45:33.651305 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:33.651273    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:45:42.453922 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.453882    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:45:42.456434 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.454177    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" podUID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" containerName="maas-api" containerID="cri-o://e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c" gracePeriod=30
Sep 06 10:45:42.716023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.715958    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:45:42.759365 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.759330    2573 generic.go:356] "Generic (PLEG): container finished" podID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" containerID="e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c" exitCode=0
Sep 06 10:45:42.759534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.759372    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"
Sep 06 10:45:42.759534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.759374    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" event={"ID":"8b3f86be-ea8e-48f8-a682-a26cf2a922d8","Type":"ContainerDied","Data":"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"}
Sep 06 10:45:42.759534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.759483    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6" event={"ID":"8b3f86be-ea8e-48f8-a682-a26cf2a922d8","Type":"ContainerDied","Data":"0f660b37a9ebb67d54e1580c6336c1e1347f850459d3c77a725cc2f9b5379832"}
Sep 06 10:45:42.759534 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.759504    2573 scope.go:117] "RemoveContainer" containerID="e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"
Sep 06 10:45:42.768695 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.768676    2573 scope.go:117] "RemoveContainer" containerID="e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"
Sep 06 10:45:42.768982 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:45:42.768964    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c\": container with ID starting with e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c not found: ID does not exist" containerID="e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"
Sep 06 10:45:42.769069 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.768994    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c"} err="failed to get container status \"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c\": rpc error: code = NotFound desc = could not find container \"e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c\": container with ID starting with e121ff88229b5faa5d8162991dfc649cc3ed663fb00c2acb831987dd144bcd3c not found: ID does not exist"
Sep 06 10:45:42.779317 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779298    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs\") pod \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") "
Sep 06 10:45:42.779412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779335    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca\") pod \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") "
Sep 06 10:45:42.779412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779390    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtvk9\" (UniqueName: \"kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9\") pod \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") "
Sep 06 10:45:42.779532 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779515    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls\") pod \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\" (UID: \"8b3f86be-ea8e-48f8-a682-a26cf2a922d8\") "
Sep 06 10:45:42.779705 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779684    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b3f86be-ea8e-48f8-a682-a26cf2a922d8" (UID: "8b3f86be-ea8e-48f8-a682-a26cf2a922d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:45:42.779852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.779836    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:42.781593 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.781564    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b3f86be-ea8e-48f8-a682-a26cf2a922d8" (UID: "8b3f86be-ea8e-48f8-a682-a26cf2a922d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:42.781681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.781591    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b3f86be-ea8e-48f8-a682-a26cf2a922d8" (UID: "8b3f86be-ea8e-48f8-a682-a26cf2a922d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:45:42.781681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.781602    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9" (OuterVolumeSpecName: "kube-api-access-gtvk9") pod "8b3f86be-ea8e-48f8-a682-a26cf2a922d8" (UID: "8b3f86be-ea8e-48f8-a682-a26cf2a922d8"). InnerVolumeSpecName "kube-api-access-gtvk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:45:42.880517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.880490    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:42.880517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.880514    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:42.880517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:42.880522    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtvk9\" (UniqueName: \"kubernetes.io/projected/8b3f86be-ea8e-48f8-a682-a26cf2a922d8-kube-api-access-gtvk9\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:45:43.080184 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:43.080132    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:45:43.084636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:43.084608    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-636b04-7964cd7979-jk9k6"]
Sep 06 10:45:43.353858 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:43.353758    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" path="/var/lib/kubelet/pods/8b3f86be-ea8e-48f8-a682-a26cf2a922d8/volumes"
Sep 06 10:45:54.025321 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025289    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:45:54.025695 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025682    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" containerName="maas-api"
Sep 06 10:45:54.025738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025698    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" containerName="maas-api"
Sep 06 10:45:54.025738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025713    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" containerName="main"
Sep 06 10:45:54.025738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025721    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" containerName="main"
Sep 06 10:45:54.025738 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025735    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" containerName="maas-api"
Sep 06 10:45:54.025946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025742    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" containerName="maas-api"
Sep 06 10:45:54.025946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025811    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="75d26e42-a741-4c6a-8ab6-b2778ae8cdb1" containerName="maas-api"
Sep 06 10:45:54.025946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025820    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b3f86be-ea8e-48f8-a682-a26cf2a922d8" containerName="maas-api"
Sep 06 10:45:54.025946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.025827    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecf32cb3-9bc9-4917-a0d3-d223723c910a" containerName="main"
Sep 06 10:45:54.030383 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.030361    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.032270 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.032242    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2eb161-serving-cert\""
Sep 06 10:45:54.037718 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.037693    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:45:54.192952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.192910    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.192952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.192955    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.193245 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.192976    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg45g\" (UniqueName: \"kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.193245 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.193105    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.294292 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.294148    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.294485 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.294290    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.294485 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.294339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.294485 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.294368    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg45g\" (UniqueName: \"kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.295054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.295027    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.296835 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.296796    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.296948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.296847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.301975 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.301948    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg45g\" (UniqueName: \"kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g\") pod \"maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.342798 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.342757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:54.621399 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.621372    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:45:54.623891 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:45:54.623859    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79674871_ab0d_44d4_ab42_30a6487742f7.slice/crio-0119ad3ef8f009ccf8d0bb2c8d4f0d22edeebe128042d69cd6790bd1ef2433d8 WatchSource:0}: Error finding container 0119ad3ef8f009ccf8d0bb2c8d4f0d22edeebe128042d69cd6790bd1ef2433d8: Status 404 returned error can't find the container with id 0119ad3ef8f009ccf8d0bb2c8d4f0d22edeebe128042d69cd6790bd1ef2433d8
Sep 06 10:45:54.659575 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.659545    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:45:54.664198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.664179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.666327 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.666306    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2eb161-dockercfg-kzk2j\""
Sep 06 10:45:54.666455 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.666307    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2eb161\""
Sep 06 10:45:54.673607 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.673577    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:45:54.799541 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.799496    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.799710 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.799558    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbqzr\" (UniqueName: \"kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.821152 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.821117    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" event={"ID":"79674871-ab0d-44d4-ab42-30a6487742f7","Type":"ContainerStarted","Data":"0119ad3ef8f009ccf8d0bb2c8d4f0d22edeebe128042d69cd6790bd1ef2433d8"}
Sep 06 10:45:54.900557 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.900519    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.900752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.900562    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbqzr\" (UniqueName: \"kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.901331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.901301    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.907451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.907430    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbqzr\" (UniqueName: \"kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:54.979045 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:54.979010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:55.250501 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.250469    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:45:55.252882 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:45:55.252856    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb539c40d_3a50_44f8_a8be_f21e123c8aca.slice/crio-b55b2894580934f76a514513f2159801d3e0e12449f960e07f34315f9d2901de WatchSource:0}: Error finding container b55b2894580934f76a514513f2159801d3e0e12449f960e07f34315f9d2901de: Status 404 returned error can't find the container with id b55b2894580934f76a514513f2159801d3e0e12449f960e07f34315f9d2901de
Sep 06 10:45:55.827811 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.827769    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" event={"ID":"b539c40d-3a50-44f8-a8be-f21e123c8aca","Type":"ContainerStarted","Data":"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"}
Sep 06 10:45:55.827994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.827844    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:45:55.827994 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.827870    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" event={"ID":"b539c40d-3a50-44f8-a8be-f21e123c8aca","Type":"ContainerStarted","Data":"b55b2894580934f76a514513f2159801d3e0e12449f960e07f34315f9d2901de"}
Sep 06 10:45:55.829421 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.829389    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" event={"ID":"79674871-ab0d-44d4-ab42-30a6487742f7","Type":"ContainerStarted","Data":"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"}
Sep 06 10:45:55.829564 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.829523    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:45:55.841407 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.841354    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" podStartSLOduration=1.841337499 podStartE2EDuration="1.841337499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:55.840937546 +0000 UTC m=+939.010787421" watchObservedRunningTime="2026-09-06 10:45:55.841337499 +0000 UTC m=+939.011187372"
Sep 06 10:45:55.854914 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:45:55.854854    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" podStartSLOduration=1.424340399 podStartE2EDuration="1.854832475s" totalImagesPullingTime="430.492076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:45:55.854213869 +0000 UTC m=+939.024063740" watchObservedRunningTime="2026-09-06 10:45:55.854832475 +0000 UTC m=+939.024682349"
Sep 06 10:46:01.839662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:01.839630    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:46:02.675464 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.675432    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:46:02.675999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.675965    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" podUID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" containerName="maas-api" containerID="cri-o://fc4ce8acb7da1008c286d6643651c1cf4d57bc10557a3fe6838616d78edff573" gracePeriod=30
Sep 06 10:46:02.716631 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.716599    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:46:02.716849 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.716823    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" podUID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" containerName="payload-pre-processing" containerID="cri-o://8f6f611aeb868cf13edcbf56562ca68d28792d7f9b0d9232732aa178c8c1ac87" gracePeriod=30
Sep 06 10:46:02.866778 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.866738    2573 generic.go:356] "Generic (PLEG): container finished" podID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" containerID="fc4ce8acb7da1008c286d6643651c1cf4d57bc10557a3fe6838616d78edff573" exitCode=0
Sep 06 10:46:02.867268 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.866938    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" event={"ID":"60b8ee5c-aaed-4720-9f98-325b8c75cfad","Type":"ContainerDied","Data":"fc4ce8acb7da1008c286d6643651c1cf4d57bc10557a3fe6838616d78edff573"}
Sep 06 10:46:02.870060 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.869911    2573 generic.go:356] "Generic (PLEG): container finished" podID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" containerID="8f6f611aeb868cf13edcbf56562ca68d28792d7f9b0d9232732aa178c8c1ac87" exitCode=0
Sep 06 10:46:02.870060 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.869960    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" event={"ID":"14c0d11e-9752-4bc3-a814-5e1d205aa74f","Type":"ContainerDied","Data":"8f6f611aeb868cf13edcbf56562ca68d28792d7f9b0d9232732aa178c8c1ac87"}
Sep 06 10:46:02.968752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.968727    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:46:02.971143 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.971117    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca\") pod \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") "
Sep 06 10:46:02.971299 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.971178    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") pod \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") "
Sep 06 10:46:02.971299 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.971256    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lctxr\" (UniqueName: \"kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr\") pod \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") "
Sep 06 10:46:02.971505 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.971475    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs\") pod \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\" (UID: \"60b8ee5c-aaed-4720-9f98-325b8c75cfad\") "
Sep 06 10:46:02.971797 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.971473    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "60b8ee5c-aaed-4720-9f98-325b8c75cfad" (UID: "60b8ee5c-aaed-4720-9f98-325b8c75cfad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:46:02.973593 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.973568    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "60b8ee5c-aaed-4720-9f98-325b8c75cfad" (UID: "60b8ee5c-aaed-4720-9f98-325b8c75cfad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:46:02.973780 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.973751    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "60b8ee5c-aaed-4720-9f98-325b8c75cfad" (UID: "60b8ee5c-aaed-4720-9f98-325b8c75cfad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:46:02.973883 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.973836    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr" (OuterVolumeSpecName: "kube-api-access-lctxr") pod "60b8ee5c-aaed-4720-9f98-325b8c75cfad" (UID: "60b8ee5c-aaed-4720-9f98-325b8c75cfad"). InnerVolumeSpecName "kube-api-access-lctxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:46:02.993233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:02.993202    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:46:03.072712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.072676    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpvhb\" (UniqueName: \"kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb\") pod \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") "
Sep 06 10:46:03.072884 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.072763    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume\") pod \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\" (UID: \"14c0d11e-9752-4bc3-a814-5e1d205aa74f\") "
Sep 06 10:46:03.072980 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.072963    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60b8ee5c-aaed-4720-9f98-325b8c75cfad-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.073019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.072988    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.073019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.073001    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lctxr\" (UniqueName: \"kubernetes.io/projected/60b8ee5c-aaed-4720-9f98-325b8c75cfad-kube-api-access-lctxr\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.073019 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.073013    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60b8ee5c-aaed-4720-9f98-325b8c75cfad-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.073146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.073129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14c0d11e-9752-4bc3-a814-5e1d205aa74f" (UID: "14c0d11e-9752-4bc3-a814-5e1d205aa74f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:46:03.074860 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.074837    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb" (OuterVolumeSpecName: "kube-api-access-lpvhb") pod "14c0d11e-9752-4bc3-a814-5e1d205aa74f" (UID: "14c0d11e-9752-4bc3-a814-5e1d205aa74f"). InnerVolumeSpecName "kube-api-access-lpvhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:46:03.173467 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.173429    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c0d11e-9752-4bc3-a814-5e1d205aa74f-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.173467 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.173461    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpvhb\" (UniqueName: \"kubernetes.io/projected/14c0d11e-9752-4bc3-a814-5e1d205aa74f-kube-api-access-lpvhb\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:46:03.875805 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.875774    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"
Sep 06 10:46:03.876294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.875821    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57" event={"ID":"60b8ee5c-aaed-4720-9f98-325b8c75cfad","Type":"ContainerDied","Data":"6524eb5b8e6d0288c9746def0f11ef27f431b3def017aa44cce4a6ad82dc541b"}
Sep 06 10:46:03.876294 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.875887    2573 scope.go:117] "RemoveContainer" containerID="fc4ce8acb7da1008c286d6643651c1cf4d57bc10557a3fe6838616d78edff573"
Sep 06 10:46:03.877360 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.877331    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh" event={"ID":"14c0d11e-9752-4bc3-a814-5e1d205aa74f","Type":"ContainerDied","Data":"47c7c8d091144f251dd12583a8f335a57851f733e4f59a7c704c5d0f247294b9"}
Sep 06 10:46:03.877620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.877599    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"
Sep 06 10:46:03.884911 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.884889    2573 scope.go:117] "RemoveContainer" containerID="8f6f611aeb868cf13edcbf56562ca68d28792d7f9b0d9232732aa178c8c1ac87"
Sep 06 10:46:03.895789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.895766    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:46:03.899945 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.899925    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2313d276-595c6754c8-gjsgh"]
Sep 06 10:46:03.909206 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.909184    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:46:03.912980 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:03.912963    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2313d276-5f4864898f-d2g57"]
Sep 06 10:46:05.352872 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:05.352839    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" path="/var/lib/kubelet/pods/14c0d11e-9752-4bc3-a814-5e1d205aa74f/volumes"
Sep 06 10:46:05.353276 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:05.353231    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" path="/var/lib/kubelet/pods/60b8ee5c-aaed-4720-9f98-325b8c75cfad/volumes"
Sep 06 10:46:06.836697 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:06.836666    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:46:22.335526 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.335491    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:46:22.336120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336068    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" containerName="payload-pre-processing"
Sep 06 10:46:22.336120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336090    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" containerName="payload-pre-processing"
Sep 06 10:46:22.336120 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336118    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" containerName="maas-api"
Sep 06 10:46:22.336337 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336126    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" containerName="maas-api"
Sep 06 10:46:22.336337 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336260    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b8ee5c-aaed-4720-9f98-325b8c75cfad" containerName="maas-api"
Sep 06 10:46:22.336337 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.336274    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c0d11e-9752-4bc3-a814-5e1d205aa74f" containerName="payload-pre-processing"
Sep 06 10:46:22.340949 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.340926    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.342993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.342965    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-eb4b45-serving-cert\""
Sep 06 10:46:22.349204 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.349180    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:46:22.454942 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.454902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.455148 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.454980    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.455148 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.455018    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5jk8\" (UniqueName: \"kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.455148 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.455048    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.556023 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.555980    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.556245 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.556041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.556245 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.556063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5jk8\" (UniqueName: \"kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.556245 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.556092    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.556245 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:22.556133    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-serving-cert: secret "maas-api-e2e-shared-b-w3-eb4b45-serving-cert" not found
Sep 06 10:46:22.556394 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:22.556251    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls podName:6c9c64c2-6051-41bd-9e23-66c0f23f102f nodeName:}" failed. No retries permitted until 2026-09-06 10:46:23.056231797 +0000 UTC m=+966.226081653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f") : secret "maas-api-e2e-shared-b-w3-eb4b45-serving-cert" not found
Sep 06 10:46:22.556780 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.556762    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.558736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.558719    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:22.566185 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:22.566137    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5jk8\" (UniqueName: \"kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:23.061588 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:23.061552    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:23.061770 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:23.061717    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-serving-cert: secret "maas-api-e2e-shared-b-w3-eb4b45-serving-cert" not found
Sep 06 10:46:23.061826 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:23.061782    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls podName:6c9c64c2-6051-41bd-9e23-66c0f23f102f nodeName:}" failed. No retries permitted until 2026-09-06 10:46:24.061764745 +0000 UTC m=+967.231614595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls") pod "maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f") : secret "maas-api-e2e-shared-b-w3-eb4b45-serving-cert" not found
Sep 06 10:46:24.072109 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.072078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:24.074709 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.074681    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:24.152688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.152656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:24.426889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.426866    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:46:24.429095 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:46:24.429067    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c9c64c2_6051_41bd_9e23_66c0f23f102f.slice/crio-3241ea8e2396cc4b4d2f5be44971daf4a3a4bf6228e387a599726b66ceb67577 WatchSource:0}: Error finding container 3241ea8e2396cc4b4d2f5be44971daf4a3a4bf6228e387a599726b66ceb67577: Status 404 returned error can't find the container with id 3241ea8e2396cc4b4d2f5be44971daf4a3a4bf6228e387a599726b66ceb67577
Sep 06 10:46:24.430532 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.430514    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:46:24.982220 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.982185    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" event={"ID":"6c9c64c2-6051-41bd-9e23-66c0f23f102f","Type":"ContainerStarted","Data":"b172bc1ec2bef8015eccba4338f275d742b18f5a49ba405311ec325b5f5b95a4"}
Sep 06 10:46:24.982377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.982247    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" event={"ID":"6c9c64c2-6051-41bd-9e23-66c0f23f102f","Type":"ContainerStarted","Data":"3241ea8e2396cc4b4d2f5be44971daf4a3a4bf6228e387a599726b66ceb67577"}
Sep 06 10:46:24.982377 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.982285    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:24.999370 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:24.999308    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" podStartSLOduration=2.563257095 podStartE2EDuration="2.999286867s" totalImagesPullingTime="436.029772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:24.996880246 +0000 UTC m=+968.166730119" watchObservedRunningTime="2026-09-06 10:46:24.999286867 +0000 UTC m=+968.169136740"
Sep 06 10:46:26.594342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.594305    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:46:26.598649 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.598628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.601274 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.601254    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-72cb16-serving-cert\""
Sep 06 10:46:26.617137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.617104    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:46:26.705299 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.705255    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfm7\" (UniqueName: \"kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.705603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.705577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.705711 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.705683    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.705772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.705734    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.807075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.807036    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.807267 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.807118    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.807267 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.807194    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.807351 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.807276    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfm7\" (UniqueName: \"kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.807824 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.807801    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.809811 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.809788    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.809925 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.809842    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.814508 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.814484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfm7\" (UniqueName: \"kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7\") pod \"maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:26.910429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:26.910398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:27.185510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.185399    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:46:27.187994 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:46:27.187962    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod850e1370_2423_46b9_8fec_f0de15daf910.slice/crio-cb151d5b40124329fe523fefe17035e21b9d75956252a7d6c2151dd6fdc48951 WatchSource:0}: Error finding container cb151d5b40124329fe523fefe17035e21b9d75956252a7d6c2151dd6fdc48951: Status 404 returned error can't find the container with id cb151d5b40124329fe523fefe17035e21b9d75956252a7d6c2151dd6fdc48951
Sep 06 10:46:27.285428 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.285396    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:46:27.290258 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.290235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.292155 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.292129    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-72cb16-dockercfg-crw9f\""
Sep 06 10:46:27.292390 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.292371    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-72cb16\""
Sep 06 10:46:27.295982 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.295958    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:46:27.413889 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.413847    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvhnj\" (UniqueName: \"kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.414063 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.413973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.514752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.514728    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvhnj\" (UniqueName: \"kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.514866 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.514790    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.515416 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.515398    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.523081 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.523057    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvhnj\" (UniqueName: \"kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj\") pod \"payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.602660 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.602621    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:27.873388 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:27.873359    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:46:27.876102 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:46:27.876071    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0caec7b9_1f4d_401b_a2c2_ace6ed602ac9.slice/crio-e1051b3f06b5ff5daa01bfe386902ab2a3167f37b3b73797ef8bd500ad5ba1be WatchSource:0}: Error finding container e1051b3f06b5ff5daa01bfe386902ab2a3167f37b3b73797ef8bd500ad5ba1be: Status 404 returned error can't find the container with id e1051b3f06b5ff5daa01bfe386902ab2a3167f37b3b73797ef8bd500ad5ba1be
Sep 06 10:46:28.001118 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.001085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" event={"ID":"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9","Type":"ContainerStarted","Data":"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"}
Sep 06 10:46:28.001456 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.001177    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:28.001456 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.001208    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" event={"ID":"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9","Type":"ContainerStarted","Data":"e1051b3f06b5ff5daa01bfe386902ab2a3167f37b3b73797ef8bd500ad5ba1be"}
Sep 06 10:46:28.003043 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.003010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" event={"ID":"850e1370-2423-46b9-8fec-f0de15daf910","Type":"ContainerStarted","Data":"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"}
Sep 06 10:46:28.003204 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.003062    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" event={"ID":"850e1370-2423-46b9-8fec-f0de15daf910","Type":"ContainerStarted","Data":"cb151d5b40124329fe523fefe17035e21b9d75956252a7d6c2151dd6fdc48951"}
Sep 06 10:46:28.003204 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.003116    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:28.014959 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.014903    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" podStartSLOduration=1.014885162 podStartE2EDuration="1.014885162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:28.014177559 +0000 UTC m=+971.184027429" watchObservedRunningTime="2026-09-06 10:46:28.014885162 +0000 UTC m=+971.184735038"
Sep 06 10:46:28.028575 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:28.028523    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" podStartSLOduration=1.752413314 podStartE2EDuration="2.028503841s" totalImagesPullingTime="276.090527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:28.028099994 +0000 UTC m=+971.197949867" watchObservedRunningTime="2026-09-06 10:46:28.028503841 +0000 UTC m=+971.198353714"
Sep 06 10:46:30.991725 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:30.991696    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:46:34.018784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:34.018748    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:46:39.013097 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:39.013010    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:46:55.529778 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.529733    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:46:55.534552 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.534523    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.536834 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.536804    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-fbf820-serving-cert\""
Sep 06 10:46:55.545714 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.545675    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:46:55.582139 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.582095    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.582342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.582174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.582342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.582295    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.582342 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.582333    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzldf\" (UniqueName: \"kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.683331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.683279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.683331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.683336    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.683604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.683412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.683604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.683444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzldf\" (UniqueName: \"kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.683604 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:55.683502    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-serving-cert: secret "maas-api-e2e-shared-b-w2-fbf820-serving-cert" not found
Sep 06 10:46:55.683604 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:46:55.683571    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls podName:8351e03a-6eaf-489b-a470-a890b8e8d04f nodeName:}" failed. No retries permitted until 2026-09-06 10:46:56.183554213 +0000 UTC m=+999.353404064 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls") pod "maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" (UID: "8351e03a-6eaf-489b-a470-a890b8e8d04f") : secret "maas-api-e2e-shared-b-w2-fbf820-serving-cert" not found
Sep 06 10:46:55.684117 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.684098    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.686014 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.685996    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:55.691397 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:55.691375    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzldf\" (UniqueName: \"kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:56.188052 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:56.188014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:56.191301 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:56.191272    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:56.449604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:56.449512    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:56.727897 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:56.727870    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:46:56.730569 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:46:56.730539    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8351e03a_6eaf_489b_a470_a890b8e8d04f.slice/crio-5940e54957881318775dbb83f8322f4f14d578c3481a6e1863dcc4384cd76876 WatchSource:0}: Error finding container 5940e54957881318775dbb83f8322f4f14d578c3481a6e1863dcc4384cd76876: Status 404 returned error can't find the container with id 5940e54957881318775dbb83f8322f4f14d578c3481a6e1863dcc4384cd76876
Sep 06 10:46:57.149643 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:57.149610    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" event={"ID":"8351e03a-6eaf-489b-a470-a890b8e8d04f","Type":"ContainerStarted","Data":"5940e54957881318775dbb83f8322f4f14d578c3481a6e1863dcc4384cd76876"}
Sep 06 10:46:58.155830 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:58.155787    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" event={"ID":"8351e03a-6eaf-489b-a470-a890b8e8d04f","Type":"ContainerStarted","Data":"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"}
Sep 06 10:46:58.156231 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:58.155858    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:46:58.173940 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:46:58.173877    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" podStartSLOduration=2.791479962 podStartE2EDuration="3.173858556s" totalImagesPullingTime="382.378594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:46:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:46:58.172001331 +0000 UTC m=+1001.341851217" watchObservedRunningTime="2026-09-06 10:46:58.173858556 +0000 UTC m=+1001.343708485"
Sep 06 10:47:04.174374 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:04.174343    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:47:05.530522 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.530485    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:47:05.535400 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.535375    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.537456 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.537424    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"openshift-service-ca.crt\""
Sep 06 10:47:05.538415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.538188    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"auth-test-model-2eb161-kserve-self-signed-certs\""
Sep 06 10:47:05.538415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.538204    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"default-dockercfg-phkh2\""
Sep 06 10:47:05.539261 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.539240    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"kube-root-ca.crt\""
Sep 06 10:47:05.543360 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.543338    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:47:05.685700 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6sg6\" (UniqueName: \"kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.685904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685714    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.685904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685761    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.685904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685782    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.685904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685810    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.685904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.685888    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.786817 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.786720    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6sg6\" (UniqueName: \"kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.786817 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.786771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.786909    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.786961    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.786996    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787068 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.787054    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787382 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.787360    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787432 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.787379    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.787468 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.787426    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.789102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.789075    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.789620 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.789602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.794103 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.794077    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6sg6\" (UniqueName: \"kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6\") pod \"auth-test-model-2eb161-kserve-8574884bc9-79pmk\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:05.860864 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:05.860806    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:06.132500 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:06.132472    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:47:06.135443 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:47:06.135417    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05362950_aa46_4b9c_99be_0465e9f557b0.slice/crio-094c0aef4624eec18ec09d6e8b82417848f362f7d549e74419cf37cae3822911 WatchSource:0}: Error finding container 094c0aef4624eec18ec09d6e8b82417848f362f7d549e74419cf37cae3822911: Status 404 returned error can't find the container with id 094c0aef4624eec18ec09d6e8b82417848f362f7d549e74419cf37cae3822911
Sep 06 10:47:06.205655 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:06.205622    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" event={"ID":"05362950-aa46-4b9c-99be-0465e9f557b0","Type":"ContainerStarted","Data":"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"}
Sep 06 10:47:06.205851 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:06.205667    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" event={"ID":"05362950-aa46-4b9c-99be-0465e9f557b0","Type":"ContainerStarted","Data":"094c0aef4624eec18ec09d6e8b82417848f362f7d549e74419cf37cae3822911"}
Sep 06 10:47:06.221769 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:06.221718    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" podStartSLOduration=1.221697148 podStartE2EDuration="1.221697148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:47:06.218759571 +0000 UTC m=+1009.388609440" watchObservedRunningTime="2026-09-06 10:47:06.221697148 +0000 UTC m=+1009.391547019"
Sep 06 10:47:15.861621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:15.861578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:15.862093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:15.861644    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:15.874560 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:15.874531    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:16.267925 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:16.267897    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:47:17.483336 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:17.483305    2573 scope.go:117] "RemoveContainer" containerID="0b3437c83e4a582b4662b98358ac89ea96d91717ab26c5b3ecabaa0ee19a0d11"
Sep 06 10:47:37.513226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.513186    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:47:37.517093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.517069    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.519047 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.519027    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aec3c2bf\""
Sep 06 10:47:37.519198 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.519070    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aec3c2bf-dockercfg-8fg8q\""
Sep 06 10:47:37.526769 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.526719    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:47:37.589376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.589341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rgh\" (UniqueName: \"kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.589584 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.589480    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.690685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.690648    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rgh\" (UniqueName: \"kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.690882 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.690733    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.691509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.691487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.698274 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.698244    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rgh\" (UniqueName: \"kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh\") pod \"payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:37.828459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:37.828371    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:38.102904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:38.102877    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:47:38.104585 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:47:38.104559    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67afc173_96db_4aa5_b620_66edd99b1fb3.slice/crio-80efe871240624ca177941a2d6156a0e8c176cf27556c6ba83649b303f1e9c29 WatchSource:0}: Error finding container 80efe871240624ca177941a2d6156a0e8c176cf27556c6ba83649b303f1e9c29: Status 404 returned error can't find the container with id 80efe871240624ca177941a2d6156a0e8c176cf27556c6ba83649b303f1e9c29
Sep 06 10:47:38.363750 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:38.363656    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" event={"ID":"67afc173-96db-4aa5-b620-66edd99b1fb3","Type":"ContainerStarted","Data":"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"}
Sep 06 10:47:38.363750 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:38.363713    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" event={"ID":"67afc173-96db-4aa5-b620-66edd99b1fb3","Type":"ContainerStarted","Data":"80efe871240624ca177941a2d6156a0e8c176cf27556c6ba83649b303f1e9c29"}
Sep 06 10:47:38.363750 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:38.363735    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:47:38.380117 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:38.380056    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" podStartSLOduration=1.380039458 podStartE2EDuration="1.380039458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:47:38.377296939 +0000 UTC m=+1041.547146818" watchObservedRunningTime="2026-09-06 10:47:38.380039458 +0000 UTC m=+1041.549889330"
Sep 06 10:47:49.370030 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:47:49.370002    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:48:14.120903 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.120864    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:48:14.125844 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.125807    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.128906 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.128877    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"kube-root-ca.crt\""
Sep 06 10:48:14.129772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.128929    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"auth-test-model-eb4b45-kserve-self-signed-certs\""
Sep 06 10:48:14.129904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.129078    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"default-dockercfg-qnl8k\""
Sep 06 10:48:14.129973 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.129366    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"openshift-service-ca.crt\""
Sep 06 10:48:14.132558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.132537    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:48:14.151902 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.151876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.152041 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.151909    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.152041 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.151937    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnvx6\" (UniqueName: \"kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.152126 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.152049    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.152205 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.152189    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.152246 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.152220    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253487 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253450    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnvx6\" (UniqueName: \"kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253686 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253497    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253686 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253561    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253686 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253724    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.253869 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.253775    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.254022 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.254003    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.254059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.254014    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.254095 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.254068    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.256121 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.256098    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.256293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.256279    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.261146 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.261125    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnvx6\" (UniqueName: \"kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6\") pod \"auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.438774 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.438744    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:14.711197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:14.711141    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:48:14.713205 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:48:14.713156    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a831cc8_ccbe_4cf3_902a_af964194ce58.slice/crio-435016b87b2caf728dc5274d7a2188a2a97eacc23c957c30c6d39bf6e9726f7c WatchSource:0}: Error finding container 435016b87b2caf728dc5274d7a2188a2a97eacc23c957c30c6d39bf6e9726f7c: Status 404 returned error can't find the container with id 435016b87b2caf728dc5274d7a2188a2a97eacc23c957c30c6d39bf6e9726f7c
Sep 06 10:48:15.545238 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:15.545204    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" event={"ID":"4a831cc8-ccbe-4cf3-902a-af964194ce58","Type":"ContainerStarted","Data":"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"}
Sep 06 10:48:15.545654 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:15.545248    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" event={"ID":"4a831cc8-ccbe-4cf3-902a-af964194ce58","Type":"ContainerStarted","Data":"435016b87b2caf728dc5274d7a2188a2a97eacc23c957c30c6d39bf6e9726f7c"}
Sep 06 10:48:15.560561 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:15.560516    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" podStartSLOduration=1.5605015070000001 podStartE2EDuration="1.560501507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:48:15.558790071 +0000 UTC m=+1078.728639943" watchObservedRunningTime="2026-09-06 10:48:15.560501507 +0000 UTC m=+1078.730351401"
Sep 06 10:48:24.438881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:24.438835    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:24.438881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:24.438886    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:24.451523 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:24.451496    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:48:24.600614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:48:24.600589    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:49:11.206688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.206650    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:49:11.207285 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.206895    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" podUID="67afc173-96db-4aa5-b620-66edd99b1fb3" containerName="payload-pre-processing" containerID="cri-o://a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0" gracePeriod=30
Sep 06 10:49:11.465032 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.464969    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:49:11.635536 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.635504    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64rgh\" (UniqueName: \"kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh\") pod \"67afc173-96db-4aa5-b620-66edd99b1fb3\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") "
Sep 06 10:49:11.635754 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.635611    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume\") pod \"67afc173-96db-4aa5-b620-66edd99b1fb3\" (UID: \"67afc173-96db-4aa5-b620-66edd99b1fb3\") "
Sep 06 10:49:11.635996 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.635970    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67afc173-96db-4aa5-b620-66edd99b1fb3" (UID: "67afc173-96db-4aa5-b620-66edd99b1fb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:49:11.637891 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.637864    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh" (OuterVolumeSpecName: "kube-api-access-64rgh") pod "67afc173-96db-4aa5-b620-66edd99b1fb3" (UID: "67afc173-96db-4aa5-b620-66edd99b1fb3"). InnerVolumeSpecName "kube-api-access-64rgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:49:11.737438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.737348    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rgh\" (UniqueName: \"kubernetes.io/projected/67afc173-96db-4aa5-b620-66edd99b1fb3-kube-api-access-64rgh\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:49:11.737438 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.737381    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67afc173-96db-4aa5-b620-66edd99b1fb3-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:49:11.829291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.829248    2573 generic.go:356] "Generic (PLEG): container finished" podID="67afc173-96db-4aa5-b620-66edd99b1fb3" containerID="a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0" exitCode=0
Sep 06 10:49:11.829459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.829298    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"
Sep 06 10:49:11.829459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.829307    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" event={"ID":"67afc173-96db-4aa5-b620-66edd99b1fb3","Type":"ContainerDied","Data":"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"}
Sep 06 10:49:11.829459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.829368    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5" event={"ID":"67afc173-96db-4aa5-b620-66edd99b1fb3","Type":"ContainerDied","Data":"80efe871240624ca177941a2d6156a0e8c176cf27556c6ba83649b303f1e9c29"}
Sep 06 10:49:11.829459 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.829398    2573 scope.go:117] "RemoveContainer" containerID="a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"
Sep 06 10:49:11.838856 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.838835    2573 scope.go:117] "RemoveContainer" containerID="a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"
Sep 06 10:49:11.839136 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:49:11.839116    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0\": container with ID starting with a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0 not found: ID does not exist" containerID="a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"
Sep 06 10:49:11.839210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.839145    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0"} err="failed to get container status \"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0\": rpc error: code = NotFound desc = could not find container \"a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0\": container with ID starting with a0d9703b7980ca980b3ddc8245cd9cb7729842d6a2d046b5d2e31d9581c75be0 not found: ID does not exist"
Sep 06 10:49:11.849722 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.849696    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:49:11.851543 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:11.851526    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aec3c2bf-65496bb9d6-cktg5"]
Sep 06 10:49:13.353317 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:49:13.353273    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67afc173-96db-4aa5-b620-66edd99b1fb3" path="/var/lib/kubelet/pods/67afc173-96db-4aa5-b620-66edd99b1fb3/volumes"
Sep 06 10:50:25.357643 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:25.357612    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:50:26.229790 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:26.229756    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:50:39.213216 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.213149    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:50:39.213853 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.213836    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67afc173-96db-4aa5-b620-66edd99b1fb3" containerName="payload-pre-processing"
Sep 06 10:50:39.213897 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.213856    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="67afc173-96db-4aa5-b620-66edd99b1fb3" containerName="payload-pre-processing"
Sep 06 10:50:39.213987 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.213976    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="67afc173-96db-4aa5-b620-66edd99b1fb3" containerName="payload-pre-processing"
Sep 06 10:50:39.217424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.217398    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.219729 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.219706    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"sub-test-model-2eb161-kserve-self-signed-certs\""
Sep 06 10:50:39.224713 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.224689    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:50:39.304106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304072    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.304106 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304112    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.304378 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304185    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.304378 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304223    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.304378 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304247    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lt5b\" (UniqueName: \"kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.304378 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.304372    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.405702 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405667    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405729    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405749    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405776    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.405962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lt5b\" (UniqueName: \"kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.406117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406214 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.406139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.406500 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.406234    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.408208 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.408189    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.408500 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.408485    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.413563 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.413541    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lt5b\" (UniqueName: \"kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b\") pod \"sub-test-model-2eb161-kserve-59578846d9-5xrd5\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.528635 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.528549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:39.797954 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:39.797887    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:50:39.800581 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:50:39.800553    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8078e012_dc2a_4a4d_979b_240ab5951c93.slice/crio-5e48d362c38e4cbfc23c4c13bc57d553f43a6f55deeb13a456f2258bfaf99832 WatchSource:0}: Error finding container 5e48d362c38e4cbfc23c4c13bc57d553f43a6f55deeb13a456f2258bfaf99832: Status 404 returned error can't find the container with id 5e48d362c38e4cbfc23c4c13bc57d553f43a6f55deeb13a456f2258bfaf99832
Sep 06 10:50:40.253323 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.253281    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" event={"ID":"8078e012-dc2a-4a4d-979b-240ab5951c93","Type":"ContainerStarted","Data":"a7e488640029900bb9c26184ab769ac3bd9f3a601c3a53588718abe2e3a6626f"}
Sep 06 10:50:40.253709 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.253334    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" event={"ID":"8078e012-dc2a-4a4d-979b-240ab5951c93","Type":"ContainerStarted","Data":"5e48d362c38e4cbfc23c4c13bc57d553f43a6f55deeb13a456f2258bfaf99832"}
Sep 06 10:50:40.277636 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.277589    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" podStartSLOduration=1.277572793 podStartE2EDuration="1.277572793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:50:40.275716387 +0000 UTC m=+1223.445566271" watchObservedRunningTime="2026-09-06 10:50:40.277572793 +0000 UTC m=+1223.447422663"
Sep 06 10:50:40.378117 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.378081    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" podUID="05362950-aa46-4b9c-99be-0465e9f557b0" containerName="main" containerID="cri-o://ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5" gracePeriod=45
Sep 06 10:50:40.630252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.630227    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:50:40.718110 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718075    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718148    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718215    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718254    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6sg6\" (UniqueName: \"kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718331    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718369    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs\") pod \"05362950-aa46-4b9c-99be-0465e9f557b0\" (UID: \"05362950-aa46-4b9c-99be-0465e9f557b0\") "
Sep 06 10:50:40.718415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718374    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:40.718565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718455    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home" (OuterVolumeSpecName: "home") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:40.718653 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718628    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:40.718712 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718698    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:40.718756 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.718716    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:40.720581 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.720558    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm" (OuterVolumeSpecName: "dshm") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:40.720694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.720588    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:40.720694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.720650    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6" (OuterVolumeSpecName: "kube-api-access-x6sg6") pod "05362950-aa46-4b9c-99be-0465e9f557b0" (UID: "05362950-aa46-4b9c-99be-0465e9f557b0"). InnerVolumeSpecName "kube-api-access-x6sg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:40.819396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.819320    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:40.819396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.819352    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05362950-aa46-4b9c-99be-0465e9f557b0-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:40.819396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.819365    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05362950-aa46-4b9c-99be-0465e9f557b0-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:40.819396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:40.819379    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6sg6\" (UniqueName: \"kubernetes.io/projected/05362950-aa46-4b9c-99be-0465e9f557b0-kube-api-access-x6sg6\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.253016 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.252958    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" podUID="4a831cc8-ccbe-4cf3-902a-af964194ce58" containerName="main" containerID="cri-o://78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce" gracePeriod=45
Sep 06 10:50:41.259930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.259901    2573 generic.go:356] "Generic (PLEG): container finished" podID="05362950-aa46-4b9c-99be-0465e9f557b0" containerID="ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5" exitCode=0
Sep 06 10:50:41.260389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.259946    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"
Sep 06 10:50:41.260389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.259986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" event={"ID":"05362950-aa46-4b9c-99be-0465e9f557b0","Type":"ContainerDied","Data":"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"}
Sep 06 10:50:41.260389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.260036    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk" event={"ID":"05362950-aa46-4b9c-99be-0465e9f557b0","Type":"ContainerDied","Data":"094c0aef4624eec18ec09d6e8b82417848f362f7d549e74419cf37cae3822911"}
Sep 06 10:50:41.260389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.260062    2573 scope.go:117] "RemoveContainer" containerID="ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"
Sep 06 10:50:41.320680 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.320647    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:50:41.324412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.324384    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/auth-test-model-2eb161-kserve-8574884bc9-79pmk"]
Sep 06 10:50:41.353144 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.353105    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05362950-aa46-4b9c-99be-0465e9f557b0" path="/var/lib/kubelet/pods/05362950-aa46-4b9c-99be-0465e9f557b0/volumes"
Sep 06 10:50:41.374580 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.374554    2573 scope.go:117] "RemoveContainer" containerID="ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"
Sep 06 10:50:41.374925 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:50:41.374904    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5\": container with ID starting with ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5 not found: ID does not exist" containerID="ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"
Sep 06 10:50:41.375007 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.374937    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5"} err="failed to get container status \"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5\": rpc error: code = NotFound desc = could not find container \"ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5\": container with ID starting with ea20533d634fc0a2223929549b75710972ef3103491dae86d176a783d282d1e5 not found: ID does not exist"
Sep 06 10:50:41.511879 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.511820    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:50:41.627604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627576    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627643    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627705    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnvx6\" (UniqueName: \"kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627727    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627781 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627747    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627974 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627784    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs\") pod \"4a831cc8-ccbe-4cf3-902a-af964194ce58\" (UID: \"4a831cc8-ccbe-4cf3-902a-af964194ce58\") "
Sep 06 10:50:41.627974 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627894    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:41.628082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.627992    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:41.628082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.628003    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home" (OuterVolumeSpecName: "home") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:41.628196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.628103    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.628196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.628119    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.628196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.628127    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.630116 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.630088    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:50:41.630252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.630129    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm" (OuterVolumeSpecName: "dshm") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:50:41.630252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.630148    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6" (OuterVolumeSpecName: "kube-api-access-lnvx6") pod "4a831cc8-ccbe-4cf3-902a-af964194ce58" (UID: "4a831cc8-ccbe-4cf3-902a-af964194ce58"). InnerVolumeSpecName "kube-api-access-lnvx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:50:41.728708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.728673    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnvx6\" (UniqueName: \"kubernetes.io/projected/4a831cc8-ccbe-4cf3-902a-af964194ce58-kube-api-access-lnvx6\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.728708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.728705    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a831cc8-ccbe-4cf3-902a-af964194ce58-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:41.728708 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:41.728716    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a831cc8-ccbe-4cf3-902a-af964194ce58-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:50:42.267531 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.267494    2573 generic.go:356] "Generic (PLEG): container finished" podID="4a831cc8-ccbe-4cf3-902a-af964194ce58" containerID="78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce" exitCode=0
Sep 06 10:50:42.268005 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.267548    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"
Sep 06 10:50:42.268005 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.267575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" event={"ID":"4a831cc8-ccbe-4cf3-902a-af964194ce58","Type":"ContainerDied","Data":"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"}
Sep 06 10:50:42.268005 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.267618    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx" event={"ID":"4a831cc8-ccbe-4cf3-902a-af964194ce58","Type":"ContainerDied","Data":"435016b87b2caf728dc5274d7a2188a2a97eacc23c957c30c6d39bf6e9726f7c"}
Sep 06 10:50:42.268005 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.267635    2573 scope.go:117] "RemoveContainer" containerID="78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"
Sep 06 10:50:42.277564 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.277542    2573 scope.go:117] "RemoveContainer" containerID="78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"
Sep 06 10:50:42.277882 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:50:42.277864    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce\": container with ID starting with 78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce not found: ID does not exist" containerID="78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"
Sep 06 10:50:42.277948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.277895    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce"} err="failed to get container status \"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce\": rpc error: code = NotFound desc = could not find container \"78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce\": container with ID starting with 78e7b9fbf23bd9117f55bb1302a850de635da96c6f636dc19b19396a5cb43dce not found: ID does not exist"
Sep 06 10:50:42.288319 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.288264    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:50:42.290027 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:42.290007    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/auth-test-model-eb4b45-kserve-7dffb48cf5-j54kx"]
Sep 06 10:50:43.352466 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:43.352438    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a831cc8-ccbe-4cf3-902a-af964194ce58" path="/var/lib/kubelet/pods/4a831cc8-ccbe-4cf3-902a-af964194ce58/volumes"
Sep 06 10:50:49.529629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:49.529592    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:49.530024 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:49.529661    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:49.541852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:49.541829    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:50:50.318665 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:50:50.318638    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:51:00.677547 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.677515    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:51:00.677920 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.677912    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a831cc8-ccbe-4cf3-902a-af964194ce58" containerName="main"
Sep 06 10:51:00.677965 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.677923    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a831cc8-ccbe-4cf3-902a-af964194ce58" containerName="main"
Sep 06 10:51:00.677965 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.677944    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05362950-aa46-4b9c-99be-0465e9f557b0" containerName="main"
Sep 06 10:51:00.677965 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.677949    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="05362950-aa46-4b9c-99be-0465e9f557b0" containerName="main"
Sep 06 10:51:00.678054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.678014    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a831cc8-ccbe-4cf3-902a-af964194ce58" containerName="main"
Sep 06 10:51:00.678054 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.678023    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="05362950-aa46-4b9c-99be-0465e9f557b0" containerName="main"
Sep 06 10:51:00.681119 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.681103    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.683096 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.683073    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-866a8b-dockercfg-x9rqp\""
Sep 06 10:51:00.683235 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.683078    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-866a8b\""
Sep 06 10:51:00.690763 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.690740    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:51:00.811343 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.811310    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjpjr\" (UniqueName: \"kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.811507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.811363    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.912660 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.912628    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjpjr\" (UniqueName: \"kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.912837 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.912686    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.913420 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.913403    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.919881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.919859    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjpjr\" (UniqueName: \"kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr\") pod \"payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:00.991538 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:00.991464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:01.261427 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:01.261400    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:51:01.263221 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:51:01.263190    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb23f28_60ac_4908_a3ae_3b178b10960c.slice/crio-9c34b6ab9f82f5f18ead7b5687f1fcbfb68067cbb5cca1704587aac57b3f6a9c WatchSource:0}: Error finding container 9c34b6ab9f82f5f18ead7b5687f1fcbfb68067cbb5cca1704587aac57b3f6a9c: Status 404 returned error can't find the container with id 9c34b6ab9f82f5f18ead7b5687f1fcbfb68067cbb5cca1704587aac57b3f6a9c
Sep 06 10:51:01.362735 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:01.362698    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" event={"ID":"3cb23f28-60ac-4908-a3ae-3b178b10960c","Type":"ContainerStarted","Data":"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"}
Sep 06 10:51:01.362903 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:01.362763    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" event={"ID":"3cb23f28-60ac-4908-a3ae-3b178b10960c","Type":"ContainerStarted","Data":"9c34b6ab9f82f5f18ead7b5687f1fcbfb68067cbb5cca1704587aac57b3f6a9c"}
Sep 06 10:51:01.362903 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:01.362798    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:01.378244 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:01.378152    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" podStartSLOduration=1.378134924 podStartE2EDuration="1.378134924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:51:01.377027767 +0000 UTC m=+1244.546877639" watchObservedRunningTime="2026-09-06 10:51:01.378134924 +0000 UTC m=+1244.547984795"
Sep 06 10:51:12.370043 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:12.370012    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:51:16.409788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.409750    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:51:16.413319 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.413298    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.415356 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.415337    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"kube-root-ca.crt\""
Sep 06 10:51:16.416093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.416069    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"openshift-service-ca.crt\""
Sep 06 10:51:16.416212 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.416094    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"sub-test-model-eb4b45-kserve-self-signed-certs\""
Sep 06 10:51:16.416212 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.416140    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"default-dockercfg-qnl8k\""
Sep 06 10:51:16.422735 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.422715    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:51:16.463282 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.463458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.463458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.463458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463418    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.463458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463452    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.463609 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.463479    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6vzn\" (UniqueName: \"kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565110 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565128    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565219    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565254    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565332 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565281    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6vzn\" (UniqueName: \"kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565584 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565524    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565598    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.565639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.565621    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.567369 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.567350    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.567681 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.567664    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.572396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.572374    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6vzn\" (UniqueName: \"kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn\") pod \"sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.724076 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.723985    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:16.996787 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:16.996764    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:51:17.001405 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:51:17.000077    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6bb1394f_eb87_4970_bcb9_b04ca9cd392f.slice/crio-4e2bd686abf6a372887c2404b1ad441a2a532b478b889a09008103836b727770 WatchSource:0}: Error finding container 4e2bd686abf6a372887c2404b1ad441a2a532b478b889a09008103836b727770: Status 404 returned error can't find the container with id 4e2bd686abf6a372887c2404b1ad441a2a532b478b889a09008103836b727770
Sep 06 10:51:17.444904 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:17.444860    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" event={"ID":"6bb1394f-eb87-4970-bcb9-b04ca9cd392f","Type":"ContainerStarted","Data":"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"}
Sep 06 10:51:17.445373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:17.444916    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" event={"ID":"6bb1394f-eb87-4970-bcb9-b04ca9cd392f","Type":"ContainerStarted","Data":"4e2bd686abf6a372887c2404b1ad441a2a532b478b889a09008103836b727770"}
Sep 06 10:51:17.463484 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:17.463433    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" podStartSLOduration=1.463417573 podStartE2EDuration="1.463417573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:51:17.460882294 +0000 UTC m=+1260.630732161" watchObservedRunningTime="2026-09-06 10:51:17.463417573 +0000 UTC m=+1260.633267443"
Sep 06 10:51:26.724365 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:26.724325    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:26.724870 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:26.724381    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:26.737250 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:26.737227    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:27.508094 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:27.508063    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:51:29.930644 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:29.930611    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:51:29.935790 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:29.935771    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:29.937801 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:29.937782    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b83cfe-dockercfg-gf52l\""
Sep 06 10:51:29.937896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:29.937827    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b83cfe\""
Sep 06 10:51:29.945255 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:29.945225    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:51:30.000133 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.000101    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7j4\" (UniqueName: \"kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.000314 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.000268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.101857 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.101818    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.102040 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.101872    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7j4\" (UniqueName: \"kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.102529 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.102509    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.113301 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.113262    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7j4\" (UniqueName: \"kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4\") pod \"payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") " pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.248592 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.248508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:30.518357 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:30.518335    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:51:30.520826 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:51:30.520795    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d705008_f6b4_4d07_9d95_e045ae4bf042.slice/crio-b79cbe5237507a135625c2dea2382d2c3fd81a1f49de4f6ea53c7ef94983bac8 WatchSource:0}: Error finding container b79cbe5237507a135625c2dea2382d2c3fd81a1f49de4f6ea53c7ef94983bac8: Status 404 returned error can't find the container with id b79cbe5237507a135625c2dea2382d2c3fd81a1f49de4f6ea53c7ef94983bac8
Sep 06 10:51:31.518297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:31.518265    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" event={"ID":"3d705008-f6b4-4d07-9d95-e045ae4bf042","Type":"ContainerStarted","Data":"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"}
Sep 06 10:51:31.518698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:31.518315    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" event={"ID":"3d705008-f6b4-4d07-9d95-e045ae4bf042","Type":"ContainerStarted","Data":"b79cbe5237507a135625c2dea2382d2c3fd81a1f49de4f6ea53c7ef94983bac8"}
Sep 06 10:51:31.518698 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:31.518346    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:51:31.534059 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:31.534011    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" podStartSLOduration=2.533996271 podStartE2EDuration="2.533996271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:51:31.532585902 +0000 UTC m=+1274.702435772" watchObservedRunningTime="2026-09-06 10:51:31.533996271 +0000 UTC m=+1274.703846140"
Sep 06 10:51:42.526186 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:51:42.526127    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:52:08.617917 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:08.617818    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:52:09.512710 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:09.512679    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:52:11.392585 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.392549    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:52:11.393032 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.392853    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" podUID="3cb23f28-60ac-4908-a3ae-3b178b10960c" containerName="payload-pre-processing" containerID="cri-o://6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918" gracePeriod=30
Sep 06 10:52:11.653505 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.653427    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:52:11.729508 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.729471    2573 generic.go:356] "Generic (PLEG): container finished" podID="3cb23f28-60ac-4908-a3ae-3b178b10960c" containerID="6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918" exitCode=0
Sep 06 10:52:11.729508 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.729519    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"
Sep 06 10:52:11.729722 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.729523    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" event={"ID":"3cb23f28-60ac-4908-a3ae-3b178b10960c","Type":"ContainerDied","Data":"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"}
Sep 06 10:52:11.729722 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.729575    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9" event={"ID":"3cb23f28-60ac-4908-a3ae-3b178b10960c","Type":"ContainerDied","Data":"9c34b6ab9f82f5f18ead7b5687f1fcbfb68067cbb5cca1704587aac57b3f6a9c"}
Sep 06 10:52:11.729722 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.729608    2573 scope.go:117] "RemoveContainer" containerID="6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"
Sep 06 10:52:11.738234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.738214    2573 scope.go:117] "RemoveContainer" containerID="6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"
Sep 06 10:52:11.738494 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:52:11.738476    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918\": container with ID starting with 6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918 not found: ID does not exist" containerID="6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"
Sep 06 10:52:11.738542 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.738502    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918"} err="failed to get container status \"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918\": rpc error: code = NotFound desc = could not find container \"6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918\": container with ID starting with 6c1bf88a174b284ea89fca55d8bb3dbd82e727778a9a42d82cc9845b3c09c918 not found: ID does not exist"
Sep 06 10:52:11.806967 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.806934    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume\") pod \"3cb23f28-60ac-4908-a3ae-3b178b10960c\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") "
Sep 06 10:52:11.807102 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.806993    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjpjr\" (UniqueName: \"kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr\") pod \"3cb23f28-60ac-4908-a3ae-3b178b10960c\" (UID: \"3cb23f28-60ac-4908-a3ae-3b178b10960c\") "
Sep 06 10:52:11.807415 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.807387    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cb23f28-60ac-4908-a3ae-3b178b10960c" (UID: "3cb23f28-60ac-4908-a3ae-3b178b10960c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:52:11.809327 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.809308    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr" (OuterVolumeSpecName: "kube-api-access-tjpjr") pod "3cb23f28-60ac-4908-a3ae-3b178b10960c" (UID: "3cb23f28-60ac-4908-a3ae-3b178b10960c"). InnerVolumeSpecName "kube-api-access-tjpjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:11.908716 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.908639    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cb23f28-60ac-4908-a3ae-3b178b10960c-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:11.908716 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:11.908667    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjpjr\" (UniqueName: \"kubernetes.io/projected/3cb23f28-60ac-4908-a3ae-3b178b10960c-kube-api-access-tjpjr\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:12.049445 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:12.049410    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:52:12.052551 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:12.052527    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-866a8b-5d7bcfc7c8-sfxb9"]
Sep 06 10:52:13.353429 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:13.353394    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cb23f28-60ac-4908-a3ae-3b178b10960c" path="/var/lib/kubelet/pods/3cb23f28-60ac-4908-a3ae-3b178b10960c/volumes"
Sep 06 10:52:23.639969 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:23.639922    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" podUID="8078e012-dc2a-4a4d-979b-240ab5951c93" containerName="main" containerID="cri-o://a7e488640029900bb9c26184ab769ac3bd9f3a601c3a53588718abe2e3a6626f" gracePeriod=45
Sep 06 10:52:23.789881 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:23.789843    2573 generic.go:356] "Generic (PLEG): container finished" podID="8078e012-dc2a-4a4d-979b-240ab5951c93" containerID="a7e488640029900bb9c26184ab769ac3bd9f3a601c3a53588718abe2e3a6626f" exitCode=0
Sep 06 10:52:23.790033 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:23.789923    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" event={"ID":"8078e012-dc2a-4a4d-979b-240ab5951c93","Type":"ContainerDied","Data":"a7e488640029900bb9c26184ab769ac3bd9f3a601c3a53588718abe2e3a6626f"}
Sep 06 10:52:23.893419 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:23.893402    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:52:24.032262 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032226    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032299    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lt5b\" (UniqueName: \"kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032339    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032400    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032423    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032487    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir\") pod \"8078e012-dc2a-4a4d-979b-240ab5951c93\" (UID: \"8078e012-dc2a-4a4d-979b-240ab5951c93\") "
Sep 06 10:52:24.032752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032481    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache" (OuterVolumeSpecName: "model-cache") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.032752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032643    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home" (OuterVolumeSpecName: "home") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.032752 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032708    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.032927 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032907    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.032974 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032932    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.032974 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.032945    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.034540 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.034510    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm" (OuterVolumeSpecName: "dshm") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.034540 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.034510    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:24.034703 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.034618    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b" (OuterVolumeSpecName: "kube-api-access-2lt5b") pod "8078e012-dc2a-4a4d-979b-240ab5951c93" (UID: "8078e012-dc2a-4a4d-979b-240ab5951c93"). InnerVolumeSpecName "kube-api-access-2lt5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:24.133789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.133753    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lt5b\" (UniqueName: \"kubernetes.io/projected/8078e012-dc2a-4a4d-979b-240ab5951c93-kube-api-access-2lt5b\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.133789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.133784    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8078e012-dc2a-4a4d-979b-240ab5951c93-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.133789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.133793    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8078e012-dc2a-4a4d-979b-240ab5951c93-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.533435 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.533391    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" podUID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" containerName="main" containerID="cri-o://91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be" gracePeriod=45
Sep 06 10:52:24.789522 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.789465    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:52:24.796340 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.796313    2573 generic.go:356] "Generic (PLEG): container finished" podID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" containerID="91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be" exitCode=0
Sep 06 10:52:24.796458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.796346    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"
Sep 06 10:52:24.796458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.796390    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" event={"ID":"6bb1394f-eb87-4970-bcb9-b04ca9cd392f","Type":"ContainerDied","Data":"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"}
Sep 06 10:52:24.796458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.796435    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75" event={"ID":"6bb1394f-eb87-4970-bcb9-b04ca9cd392f","Type":"ContainerDied","Data":"4e2bd686abf6a372887c2404b1ad441a2a532b478b889a09008103836b727770"}
Sep 06 10:52:24.796458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.796456    2573 scope.go:117] "RemoveContainer" containerID="91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"
Sep 06 10:52:24.797788 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.797762    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5" event={"ID":"8078e012-dc2a-4a4d-979b-240ab5951c93","Type":"ContainerDied","Data":"5e48d362c38e4cbfc23c4c13bc57d553f43a6f55deeb13a456f2258bfaf99832"}
Sep 06 10:52:24.797896 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.797807    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"
Sep 06 10:52:24.805851 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.805821    2573 scope.go:117] "RemoveContainer" containerID="91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"
Sep 06 10:52:24.807030 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:52:24.807001    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be\": container with ID starting with 91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be not found: ID does not exist" containerID="91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"
Sep 06 10:52:24.807234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.807041    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be"} err="failed to get container status \"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be\": rpc error: code = NotFound desc = could not find container \"91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be\": container with ID starting with 91a92768ba19260e8df0500e4964d54bbb0cf4b53e7ce1208fff18f021e111be not found: ID does not exist"
Sep 06 10:52:24.807234 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.807064    2573 scope.go:117] "RemoveContainer" containerID="a7e488640029900bb9c26184ab769ac3bd9f3a601c3a53588718abe2e3a6626f"
Sep 06 10:52:24.820390 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.820369    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:52:24.823799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.823777    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/sub-test-model-2eb161-kserve-59578846d9-5xrd5"]
Sep 06 10:52:24.942086 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942059    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942281 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942103    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942281 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942130    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6vzn\" (UniqueName: \"kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942281 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942192    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942281 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942236    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942337    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.942530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942352    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache" (OuterVolumeSpecName: "model-cache") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.942530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942351    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm\") pod \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\" (UID: \"6bb1394f-eb87-4970-bcb9-b04ca9cd392f\") "
Sep 06 10:52:24.942530 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942453    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home" (OuterVolumeSpecName: "home") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.942735 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942718    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.942775 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942742    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.942775 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.942755    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:24.944511 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.944481    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:24.944511 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.944494    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm" (OuterVolumeSpecName: "dshm") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:52:24.944639 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:24.944524    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn" (OuterVolumeSpecName: "kube-api-access-s6vzn") pod "6bb1394f-eb87-4970-bcb9-b04ca9cd392f" (UID: "6bb1394f-eb87-4970-bcb9-b04ca9cd392f"). InnerVolumeSpecName "kube-api-access-s6vzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:25.043758 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.043679    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:25.043758 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.043704    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6vzn\" (UniqueName: \"kubernetes.io/projected/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-kube-api-access-s6vzn\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:25.043758 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.043715    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bb1394f-eb87-4970-bcb9-b04ca9cd392f-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:25.115706 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.115672    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:52:25.118805 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.118785    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/sub-test-model-eb4b45-kserve-fd9bc8cd7-l7t75"]
Sep 06 10:52:25.353480 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.353399    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" path="/var/lib/kubelet/pods/6bb1394f-eb87-4970-bcb9-b04ca9cd392f/volumes"
Sep 06 10:52:25.353785 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:25.353773    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8078e012-dc2a-4a4d-979b-240ab5951c93" path="/var/lib/kubelet/pods/8078e012-dc2a-4a4d-979b-240ab5951c93/volumes"
Sep 06 10:52:27.522331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.522302    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:52:27.523121 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523099    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8078e012-dc2a-4a4d-979b-240ab5951c93" containerName="main"
Sep 06 10:52:27.523121 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523123    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8078e012-dc2a-4a4d-979b-240ab5951c93" containerName="main"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523139    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" containerName="main"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523144    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" containerName="main"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523153    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb23f28-60ac-4908-a3ae-3b178b10960c" containerName="payload-pre-processing"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523176    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb23f28-60ac-4908-a3ae-3b178b10960c" containerName="payload-pre-processing"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523255    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb23f28-60ac-4908-a3ae-3b178b10960c" containerName="payload-pre-processing"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523270    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8078e012-dc2a-4a4d-979b-240ab5951c93" containerName="main"
Sep 06 10:52:27.523290 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.523279    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bb1394f-eb87-4970-bcb9-b04ca9cd392f" containerName="main"
Sep 06 10:52:27.527791 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.527772    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.529868 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.529843    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"openshift-service-ca.crt\""
Sep 06 10:52:27.529971 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.529934    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"kube-root-ca.crt\""
Sep 06 10:52:27.530566 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.530545    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"rate-test-model-2eb161-kserve-self-signed-certs\""
Sep 06 10:52:27.530566 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.530554    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-2eb161\"/\"default-dockercfg-phkh2\""
Sep 06 10:52:27.536440 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.536419    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:52:27.664638 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664607    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.664832 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.664832 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664725    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmlc8\" (UniqueName: \"kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.664832 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664828    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.664988 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664857    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.664988 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.664887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765576    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765614 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765617    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765645    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765750    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.765873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.765778    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmlc8\" (UniqueName: \"kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.766072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.766035    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.766072 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.766056    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.766193 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.766117    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.768065 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.768045    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.768249 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.768232    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.772960 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.772896    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmlc8\" (UniqueName: \"kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8\") pod \"rate-test-model-2eb161-kserve-b7bdfc94c-h6848\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") " pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:27.839227 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:27.839197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:28.108769 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:28.108746    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:52:28.110947 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:52:28.110925    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda43cf1a8_6b07_4b33_b33b_d33d9fbc900a.slice/crio-a9f17209c1846aa7a71b50038290f71f77390a263836da16cb2c058c7b0ea679 WatchSource:0}: Error finding container a9f17209c1846aa7a71b50038290f71f77390a263836da16cb2c058c7b0ea679: Status 404 returned error can't find the container with id a9f17209c1846aa7a71b50038290f71f77390a263836da16cb2c058c7b0ea679
Sep 06 10:52:28.820913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:28.820883    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" event={"ID":"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a","Type":"ContainerStarted","Data":"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"}
Sep 06 10:52:28.820913 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:28.820928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" event={"ID":"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a","Type":"ContainerStarted","Data":"a9f17209c1846aa7a71b50038290f71f77390a263836da16cb2c058c7b0ea679"}
Sep 06 10:52:28.835039 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:28.834982    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" podStartSLOduration=1.834965349 podStartE2EDuration="1.834965349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:52:28.833612349 +0000 UTC m=+1332.003462215" watchObservedRunningTime="2026-09-06 10:52:28.834965349 +0000 UTC m=+1332.004815219"
Sep 06 10:52:33.987451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:33.987408    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:52:33.987873 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:33.987656    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" podUID="3d705008-f6b4-4d07-9d95-e045ae4bf042" containerName="payload-processing" containerID="cri-o://17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4" gracePeriod=30
Sep 06 10:52:34.234793 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.234772    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:52:34.316973 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.316891    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume\") pod \"3d705008-f6b4-4d07-9d95-e045ae4bf042\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") "
Sep 06 10:52:34.316973 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.316964    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j7j4\" (UniqueName: \"kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4\") pod \"3d705008-f6b4-4d07-9d95-e045ae4bf042\" (UID: \"3d705008-f6b4-4d07-9d95-e045ae4bf042\") "
Sep 06 10:52:34.317288 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.317264    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d705008-f6b4-4d07-9d95-e045ae4bf042" (UID: "3d705008-f6b4-4d07-9d95-e045ae4bf042"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:52:34.319202 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.319179    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4" (OuterVolumeSpecName: "kube-api-access-6j7j4") pod "3d705008-f6b4-4d07-9d95-e045ae4bf042" (UID: "3d705008-f6b4-4d07-9d95-e045ae4bf042"). InnerVolumeSpecName "kube-api-access-6j7j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:34.418122 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.418084    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d705008-f6b4-4d07-9d95-e045ae4bf042-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:34.418122 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.418113    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7j4\" (UniqueName: \"kubernetes.io/projected/3d705008-f6b4-4d07-9d95-e045ae4bf042-kube-api-access-6j7j4\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:34.854871 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.854835    2573 generic.go:356] "Generic (PLEG): container finished" podID="3d705008-f6b4-4d07-9d95-e045ae4bf042" containerID="17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4" exitCode=0
Sep 06 10:52:34.855075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.854883    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"
Sep 06 10:52:34.855075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.854928    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" event={"ID":"3d705008-f6b4-4d07-9d95-e045ae4bf042","Type":"ContainerDied","Data":"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"}
Sep 06 10:52:34.855075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.854986    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds" event={"ID":"3d705008-f6b4-4d07-9d95-e045ae4bf042","Type":"ContainerDied","Data":"b79cbe5237507a135625c2dea2382d2c3fd81a1f49de4f6ea53c7ef94983bac8"}
Sep 06 10:52:34.855075 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.855012    2573 scope.go:117] "RemoveContainer" containerID="17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"
Sep 06 10:52:34.864176 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.864144    2573 scope.go:117] "RemoveContainer" containerID="17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"
Sep 06 10:52:34.864444 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:52:34.864422    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4\": container with ID starting with 17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4 not found: ID does not exist" containerID="17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"
Sep 06 10:52:34.864507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.864455    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4"} err="failed to get container status \"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4\": rpc error: code = NotFound desc = could not find container \"17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4\": container with ID starting with 17890bff69a213072e09510972424a6782ba32550776497a71a2eacdea64d6f4 not found: ID does not exist"
Sep 06 10:52:34.874237 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.874215    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:52:34.881794 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:34.881774    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b83cfe-799569dfff-fbhds"]
Sep 06 10:52:35.353682 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:35.353647    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d705008-f6b4-4d07-9d95-e045ae4bf042" path="/var/lib/kubelet/pods/3d705008-f6b4-4d07-9d95-e045ae4bf042/volumes"
Sep 06 10:52:37.839746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:37.839710    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:37.840216 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:37.839784    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:37.851930 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:37.851905    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:37.889373 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:37.889324    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:52:44.820661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.820631    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:52:44.821083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.821069    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d705008-f6b4-4d07-9d95-e045ae4bf042" containerName="payload-processing"
Sep 06 10:52:44.821145 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.821086    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d705008-f6b4-4d07-9d95-e045ae4bf042" containerName="payload-processing"
Sep 06 10:52:44.821208 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.821154    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d705008-f6b4-4d07-9d95-e045ae4bf042" containerName="payload-processing"
Sep 06 10:52:44.825695 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.825676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.827879 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.827855    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"openshift-service-ca.crt\""
Sep 06 10:52:44.828468 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.828448    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"default-dockercfg-qnl8k\""
Sep 06 10:52:44.828576 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.828546    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"kube-root-ca.crt\""
Sep 06 10:52:44.828643 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.828573    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-eb4b45\"/\"rate-test-model-eb4b45-kserve-self-signed-certs\""
Sep 06 10:52:44.835467 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.835445    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:52:44.914382 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914351    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.914576 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914395    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.914576 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914488    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.914576 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.914737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914608    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbg4w\" (UniqueName: \"kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:44.914737 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:44.914648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016032    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbg4w\" (UniqueName: \"kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016072    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016107    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016278 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016243    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016535 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016487    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016610 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016581    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.016661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.016629    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.018509 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.018484    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.018768 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.018752    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.023330 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.023311    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbg4w\" (UniqueName: \"kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w\") pod \"rate-test-model-eb4b45-kserve-d9769747f-6xd2d\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") " pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.137604 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.137508    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:45.408694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.408665    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:52:45.410079 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:52:45.410051    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6802290_f25f_4a8d_a5f9_bc3bfc9e5cd5.slice/crio-ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93 WatchSource:0}: Error finding container ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93: Status 404 returned error can't find the container with id ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93
Sep 06 10:52:45.914421 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.914385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" event={"ID":"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5","Type":"ContainerStarted","Data":"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"}
Sep 06 10:52:45.914815 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.914438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" event={"ID":"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5","Type":"ContainerStarted","Data":"ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93"}
Sep 06 10:52:45.930982 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:45.930930    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" podStartSLOduration=1.9309134559999999 podStartE2EDuration="1.930913456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:52:45.929107483 +0000 UTC m=+1349.098957377" watchObservedRunningTime="2026-09-06 10:52:45.930913456 +0000 UTC m=+1349.100763327"
Sep 06 10:52:51.153983 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.153950    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:52:51.154579 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.154192    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" podUID="850e1370-2423-46b9-8fec-f0de15daf910" containerName="maas-api" containerID="cri-o://91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3" gracePeriod=30
Sep 06 10:52:51.216192 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.216127    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:52:51.216691 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.216615    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" podUID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" containerName="payload-pre-processing" containerID="cri-o://4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c" gracePeriod=30
Sep 06 10:52:51.473654 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.473631    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:52:51.497406 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.497121    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:52:51.584204 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584157    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume\") pod \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") "
Sep 06 10:52:51.584394 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584216    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs\") pod \"850e1370-2423-46b9-8fec-f0de15daf910\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") "
Sep 06 10:52:51.584394 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584354    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdfm7\" (UniqueName: \"kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7\") pod \"850e1370-2423-46b9-8fec-f0de15daf910\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") "
Sep 06 10:52:51.584507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584403    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvhnj\" (UniqueName: \"kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj\") pod \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\" (UID: \"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9\") "
Sep 06 10:52:51.584507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584435    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca\") pod \"850e1370-2423-46b9-8fec-f0de15daf910\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") "
Sep 06 10:52:51.584507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584479    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls\") pod \"850e1370-2423-46b9-8fec-f0de15daf910\" (UID: \"850e1370-2423-46b9-8fec-f0de15daf910\") "
Sep 06 10:52:51.584650 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584582    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" (UID: "0caec7b9-1f4d-401b-a2c2-ace6ed602ac9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:52:51.584839 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584808    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.584943 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.584910    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "850e1370-2423-46b9-8fec-f0de15daf910" (UID: "850e1370-2423-46b9-8fec-f0de15daf910"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:52:51.586652 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.586620    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7" (OuterVolumeSpecName: "kube-api-access-hdfm7") pod "850e1370-2423-46b9-8fec-f0de15daf910" (UID: "850e1370-2423-46b9-8fec-f0de15daf910"). InnerVolumeSpecName "kube-api-access-hdfm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:51.586652 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.586641    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "850e1370-2423-46b9-8fec-f0de15daf910" (UID: "850e1370-2423-46b9-8fec-f0de15daf910"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:51.586652 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.586638    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj" (OuterVolumeSpecName: "kube-api-access-vvhnj") pod "0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" (UID: "0caec7b9-1f4d-401b-a2c2-ace6ed602ac9"). InnerVolumeSpecName "kube-api-access-vvhnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:52:51.586954 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.586928    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "850e1370-2423-46b9-8fec-f0de15daf910" (UID: "850e1370-2423-46b9-8fec-f0de15daf910"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:52:51.685736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.685655    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfm7\" (UniqueName: \"kubernetes.io/projected/850e1370-2423-46b9-8fec-f0de15daf910-kube-api-access-hdfm7\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.685736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.685684    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvhnj\" (UniqueName: \"kubernetes.io/projected/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9-kube-api-access-vvhnj\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.685736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.685692    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e1370-2423-46b9-8fec-f0de15daf910-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.685736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.685700    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.685736 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.685708    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/850e1370-2423-46b9-8fec-f0de15daf910-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:52:51.944952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.944876    2573 generic.go:356] "Generic (PLEG): container finished" podID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" containerID="4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c" exitCode=0
Sep 06 10:52:51.944952 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.944922    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"
Sep 06 10:52:51.945183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.944959    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" event={"ID":"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9","Type":"ContainerDied","Data":"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"}
Sep 06 10:52:51.945183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.945010    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9" event={"ID":"0caec7b9-1f4d-401b-a2c2-ace6ed602ac9","Type":"ContainerDied","Data":"e1051b3f06b5ff5daa01bfe386902ab2a3167f37b3b73797ef8bd500ad5ba1be"}
Sep 06 10:52:51.945183 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.945037    2573 scope.go:117] "RemoveContainer" containerID="4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"
Sep 06 10:52:51.946376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.946355    2573 generic.go:356] "Generic (PLEG): container finished" podID="850e1370-2423-46b9-8fec-f0de15daf910" containerID="91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3" exitCode=0
Sep 06 10:52:51.946507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.946385    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" event={"ID":"850e1370-2423-46b9-8fec-f0de15daf910","Type":"ContainerDied","Data":"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"}
Sep 06 10:52:51.946573 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.946510    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv" event={"ID":"850e1370-2423-46b9-8fec-f0de15daf910","Type":"ContainerDied","Data":"cb151d5b40124329fe523fefe17035e21b9d75956252a7d6c2151dd6fdc48951"}
Sep 06 10:52:51.946573 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.946413    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"
Sep 06 10:52:51.955136 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.955121    2573 scope.go:117] "RemoveContainer" containerID="4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"
Sep 06 10:52:51.955476 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:52:51.955458    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c\": container with ID starting with 4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c not found: ID does not exist" containerID="4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"
Sep 06 10:52:51.955547 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.955481    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c"} err="failed to get container status \"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c\": rpc error: code = NotFound desc = could not find container \"4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c\": container with ID starting with 4617054672c41e19fe93914dffc03c922c20ffa5ed959364646087976843650c not found: ID does not exist"
Sep 06 10:52:51.955547 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.955496    2573 scope.go:117] "RemoveContainer" containerID="91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"
Sep 06 10:52:51.963687 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.963659    2573 scope.go:117] "RemoveContainer" containerID="91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"
Sep 06 10:52:51.964109 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:52:51.964089    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3\": container with ID starting with 91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3 not found: ID does not exist" containerID="91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"
Sep 06 10:52:51.964212 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.964115    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3"} err="failed to get container status \"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3\": rpc error: code = NotFound desc = could not find container \"91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3\": container with ID starting with 91cad26399454057dc31f4fba6a4e99827d50145139e43cc452afc44d62f8bb3 not found: ID does not exist"
Sep 06 10:52:51.966175 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.966137    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:52:51.968519 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.968500    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-72cb16-85599c46f4-t8xf9"]
Sep 06 10:52:51.981085 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.981054    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:52:51.983064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:51.983043    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-72cb16-b957c7db4-x25pv"]
Sep 06 10:52:53.353690 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:53.353656    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" path="/var/lib/kubelet/pods/0caec7b9-1f4d-401b-a2c2-ace6ed602ac9/volumes"
Sep 06 10:52:53.354071 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:53.354024    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="850e1370-2423-46b9-8fec-f0de15daf910" path="/var/lib/kubelet/pods/850e1370-2423-46b9-8fec-f0de15daf910/volumes"
Sep 06 10:52:55.138180 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:55.138130    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:55.138548 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:55.138196    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:55.150433 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:55.150407    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:52:55.978705 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:52:55.978673    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:53:08.580784 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.580690    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:53:08.581212 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.580958    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" podUID="8351e03a-6eaf-489b-a470-a890b8e8d04f" containerName="maas-api" containerID="cri-o://acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57" gracePeriod=30
Sep 06 10:53:08.828312 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.828280    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:53:08.953138 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953103    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzldf\" (UniqueName: \"kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf\") pod \"8351e03a-6eaf-489b-a470-a890b8e8d04f\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") "
Sep 06 10:53:08.953331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953194    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca\") pod \"8351e03a-6eaf-489b-a470-a890b8e8d04f\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") "
Sep 06 10:53:08.953331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953242    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") pod \"8351e03a-6eaf-489b-a470-a890b8e8d04f\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") "
Sep 06 10:53:08.953470 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953363    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs\") pod \"8351e03a-6eaf-489b-a470-a890b8e8d04f\" (UID: \"8351e03a-6eaf-489b-a470-a890b8e8d04f\") "
Sep 06 10:53:08.953605 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953568    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8351e03a-6eaf-489b-a470-a890b8e8d04f" (UID: "8351e03a-6eaf-489b-a470-a890b8e8d04f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:53:08.953730 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.953713    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8351e03a-6eaf-489b-a470-a890b8e8d04f-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:08.955511 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.955477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8351e03a-6eaf-489b-a470-a890b8e8d04f" (UID: "8351e03a-6eaf-489b-a470-a890b8e8d04f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:53:08.955748 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.955721    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8351e03a-6eaf-489b-a470-a890b8e8d04f" (UID: "8351e03a-6eaf-489b-a470-a890b8e8d04f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:53:08.955798 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:08.955775    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf" (OuterVolumeSpecName: "kube-api-access-lzldf") pod "8351e03a-6eaf-489b-a470-a890b8e8d04f" (UID: "8351e03a-6eaf-489b-a470-a890b8e8d04f"). InnerVolumeSpecName "kube-api-access-lzldf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:53:09.030191 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.030134    2573 generic.go:356] "Generic (PLEG): container finished" podID="8351e03a-6eaf-489b-a470-a890b8e8d04f" containerID="acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57" exitCode=0
Sep 06 10:53:09.030356 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.030214    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"
Sep 06 10:53:09.030356 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.030206    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" event={"ID":"8351e03a-6eaf-489b-a470-a890b8e8d04f","Type":"ContainerDied","Data":"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"}
Sep 06 10:53:09.030356 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.030341    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg" event={"ID":"8351e03a-6eaf-489b-a470-a890b8e8d04f","Type":"ContainerDied","Data":"5940e54957881318775dbb83f8322f4f14d578c3481a6e1863dcc4384cd76876"}
Sep 06 10:53:09.030494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.030364    2573 scope.go:117] "RemoveContainer" containerID="acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"
Sep 06 10:53:09.041465 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.041442    2573 scope.go:117] "RemoveContainer" containerID="acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"
Sep 06 10:53:09.041803 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:53:09.041783    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57\": container with ID starting with acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57 not found: ID does not exist" containerID="acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"
Sep 06 10:53:09.041861 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.041812    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57"} err="failed to get container status \"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57\": rpc error: code = NotFound desc = could not find container \"acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57\": container with ID starting with acbf1741daac773323ebff49cf091f7e17ce0e4250006a57359f93531bdddb57 not found: ID does not exist"
Sep 06 10:53:09.054211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.054184    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:53:09.054612 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.054593    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzldf\" (UniqueName: \"kubernetes.io/projected/8351e03a-6eaf-489b-a470-a890b8e8d04f-kube-api-access-lzldf\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:09.054655 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.054617    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:09.054655 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.054628    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8351e03a-6eaf-489b-a470-a890b8e8d04f-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:09.058572 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.058540    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-fbf820-74cd976566-jqlsg"]
Sep 06 10:53:09.352433 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:09.352355    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8351e03a-6eaf-489b-a470-a890b8e8d04f" path="/var/lib/kubelet/pods/8351e03a-6eaf-489b-a470-a890b8e8d04f/volumes"
Sep 06 10:53:31.915861 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:31.915826    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:53:32.839696 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:32.839653    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:53:38.754506 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754472    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754841    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8351e03a-6eaf-489b-a470-a890b8e8d04f" containerName="maas-api"
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754852    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="8351e03a-6eaf-489b-a470-a890b8e8d04f" containerName="maas-api"
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754861    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" containerName="payload-pre-processing"
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754867    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" containerName="payload-pre-processing"
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754882    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="850e1370-2423-46b9-8fec-f0de15daf910" containerName="maas-api"
Sep 06 10:53:38.754946 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754887    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="850e1370-2423-46b9-8fec-f0de15daf910" containerName="maas-api"
Sep 06 10:53:38.755131 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754969    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="850e1370-2423-46b9-8fec-f0de15daf910" containerName="maas-api"
Sep 06 10:53:38.755131 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754979    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="0caec7b9-1f4d-401b-a2c2-ace6ed602ac9" containerName="payload-pre-processing"
Sep 06 10:53:38.755131 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.754986    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="8351e03a-6eaf-489b-a470-a890b8e8d04f" containerName="maas-api"
Sep 06 10:53:38.759083 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.759052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.760993 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.760970    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-69ef21\""
Sep 06 10:53:38.761112 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.761090    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-69ef21-dockercfg-tlqvd\""
Sep 06 10:53:38.767339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.767319    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:53:38.841176 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.841138    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.841363 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.841203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7b7\" (UniqueName: \"kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.942479 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.942445    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.942653 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.942487    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7b7\" (UniqueName: \"kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.943185 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.943139    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:38.949694 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:38.949662    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7b7\" (UniqueName: \"kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7\") pod \"payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:39.070816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:39.070738    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:39.350598 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:53:39.350564    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcceb09ff_9da4_45ee_a62c_c4c84140db3c.slice/crio-41629864c02401ada2ce4323c927a688d1dc370ab4dd5be8a9ea37a7ec25d7b3 WatchSource:0}: Error finding container 41629864c02401ada2ce4323c927a688d1dc370ab4dd5be8a9ea37a7ec25d7b3: Status 404 returned error can't find the container with id 41629864c02401ada2ce4323c927a688d1dc370ab4dd5be8a9ea37a7ec25d7b3
Sep 06 10:53:39.362616 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:39.362591    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:53:40.178776 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:40.178735    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" event={"ID":"cceb09ff-9da4-45ee-a62c-c4c84140db3c","Type":"ContainerStarted","Data":"24a8dc2e613bb1057ad6446d8a25b4aa6fb6f57544d7dacc6d468a0fcf55816e"}
Sep 06 10:53:40.179283 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:40.178795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" event={"ID":"cceb09ff-9da4-45ee-a62c-c4c84140db3c","Type":"ContainerStarted","Data":"41629864c02401ada2ce4323c927a688d1dc370ab4dd5be8a9ea37a7ec25d7b3"}
Sep 06 10:53:40.179283 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:40.178874    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:53:40.193941 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:40.193889    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" podStartSLOduration=2.193875012 podStartE2EDuration="2.193875012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:53:40.192029505 +0000 UTC m=+1403.361879375" watchObservedRunningTime="2026-09-06 10:53:40.193875012 +0000 UTC m=+1403.363724882"
Sep 06 10:53:46.938931 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:46.938867    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" podUID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" containerName="main" containerID="cri-o://bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695" gracePeriod=45
Sep 06 10:53:47.201064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.200999    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:53:47.212707 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.212677    2573 generic.go:356] "Generic (PLEG): container finished" podID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" containerID="bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695" exitCode=0
Sep 06 10:53:47.212846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.212721    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"
Sep 06 10:53:47.212846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.212756    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" event={"ID":"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a","Type":"ContainerDied","Data":"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"}
Sep 06 10:53:47.212846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.212795    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848" event={"ID":"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a","Type":"ContainerDied","Data":"a9f17209c1846aa7a71b50038290f71f77390a263836da16cb2c058c7b0ea679"}
Sep 06 10:53:47.212846 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.212815    2573 scope.go:117] "RemoveContainer" containerID="bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"
Sep 06 10:53:47.224279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.224253    2573 scope.go:117] "RemoveContainer" containerID="bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"
Sep 06 10:53:47.224566 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:53:47.224545    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695\": container with ID starting with bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695 not found: ID does not exist" containerID="bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"
Sep 06 10:53:47.224648 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.224573    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695"} err="failed to get container status \"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695\": rpc error: code = NotFound desc = could not find container \"bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695\": container with ID starting with bdf92a0109ddf7122eb51f633b620fa7b74338c9d4cf93e0d30f77f7837bb695 not found: ID does not exist"
Sep 06 10:53:47.322982 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.322957    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323029    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323079    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmlc8\" (UniqueName: \"kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323111    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323124    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323211 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323150    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm\") pod \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\" (UID: \"a43cf1a8-6b07-4b33-b33b-d33d9fbc900a\") "
Sep 06 10:53:47.323469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323259    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home" (OuterVolumeSpecName: "home") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:47.323469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323312    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:47.323469 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323421    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:47.323628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323610    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.323687 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323632    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.323687 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.323668    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.325502 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.325471    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:53:47.325502 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.325485    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8" (OuterVolumeSpecName: "kube-api-access-jmlc8") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "kube-api-access-jmlc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:53:47.325651 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.325537    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm" (OuterVolumeSpecName: "dshm") pod "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" (UID: "a43cf1a8-6b07-4b33-b33b-d33d9fbc900a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:47.424628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.424581    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmlc8\" (UniqueName: \"kubernetes.io/projected/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-kube-api-access-jmlc8\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.424628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.424614    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.424628 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.424624    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:47.527508 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.527424    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:53:47.533078 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.533047    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-2eb161/rate-test-model-2eb161-kserve-b7bdfc94c-h6848"]
Sep 06 10:53:47.859759 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:47.859656    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" podUID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" containerName="main" containerID="cri-o://aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0" gracePeriod=45
Sep 06 10:53:47.898585 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:53:47.898565    2573 container.go:598] Failed to update stats for container "/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6802290_f25f_4a8d_a5f9_bc3bfc9e5cd5.slice/crio-ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93": error while statting cgroup v2: [unable to parse /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6802290_f25f_4a8d_a5f9_bc3bfc9e5cd5.slice/crio-ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93/memory.stat: read /sys/fs/cgroup/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6802290_f25f_4a8d_a5f9_bc3bfc9e5cd5.slice/crio-ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93/memory.stat: no such device], continuing to push stats
Sep 06 10:53:48.111297 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.111237    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:53:48.219236 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.219202    2573 generic.go:356] "Generic (PLEG): container finished" podID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" containerID="aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0" exitCode=0
Sep 06 10:53:48.219424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.219257    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"
Sep 06 10:53:48.219424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.219301    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" event={"ID":"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5","Type":"ContainerDied","Data":"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"}
Sep 06 10:53:48.219424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.219349    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d" event={"ID":"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5","Type":"ContainerDied","Data":"ae024eb51a194049e64730f36fbff084c4e5ecb4c00b787f973248e0429f7e93"}
Sep 06 10:53:48.219424 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.219373    2573 scope.go:117] "RemoveContainer" containerID="aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"
Sep 06 10:53:48.227510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.227491    2573 scope.go:117] "RemoveContainer" containerID="aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"
Sep 06 10:53:48.227785 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:53:48.227767    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0\": container with ID starting with aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0 not found: ID does not exist" containerID="aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"
Sep 06 10:53:48.227829 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.227795    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0"} err="failed to get container status \"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0\": rpc error: code = NotFound desc = could not find container \"aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0\": container with ID starting with aabf5a70f647449aaa7da820c065ce686a6814d61bcae4c9a2c9fd36065b9bf0 not found: ID does not exist"
Sep 06 10:53:48.233087 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233069    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233119    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233147 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233137    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbg4w\" (UniqueName: \"kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233153    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233233 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233229    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233270    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home\") pod \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\" (UID: \"f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5\") "
Sep 06 10:53:48.233391 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233353    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:48.233539 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233515    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:48.233627 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233533    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home" (OuterVolumeSpecName: "home") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:48.233719 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233702    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.233760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233728    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.233760 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.233743    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.235331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.235310    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm" (OuterVolumeSpecName: "dshm") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:53:48.235396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.235327    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w" (OuterVolumeSpecName: "kube-api-access-xbg4w") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "kube-api-access-xbg4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:53:48.235396 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.235326    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" (UID: "f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:53:48.335035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.335002    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.335035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.335029    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.335035 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.335040    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbg4w\" (UniqueName: \"kubernetes.io/projected/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5-kube-api-access-xbg4w\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:53:48.538759 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.538726    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:53:48.542402 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:48.542376    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-eb4b45/rate-test-model-eb4b45-kserve-d9769747f-6xd2d"]
Sep 06 10:53:49.353111 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:49.353080    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" path="/var/lib/kubelet/pods/a43cf1a8-6b07-4b33-b33b-d33d9fbc900a/volumes"
Sep 06 10:53:49.353506 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:49.353466    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" path="/var/lib/kubelet/pods/f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5/volumes"
Sep 06 10:53:51.185070 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:53:51.185035    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:54:47.013961 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.013926    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:54:47.014439 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014411    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" containerName="main"
Sep 06 10:54:47.014439 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014426    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" containerName="main"
Sep 06 10:54:47.014439 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014436    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" containerName="main"
Sep 06 10:54:47.014439 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014441    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" containerName="main"
Sep 06 10:54:47.014789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014504    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43cf1a8-6b07-4b33-b33b-d33d9fbc900a" containerName="main"
Sep 06 10:54:47.014789 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.014518    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6802290-f25f-4a8d-a5f9-bc3bfc9e5cd5" containerName="main"
Sep 06 10:54:47.017814 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.017797    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.020818 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.020798    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-69ef21\"/\"openshift-service-ca.crt\""
Sep 06 10:54:47.020939 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.020819    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-69ef21\"/\"default-dockercfg-dk85j\""
Sep 06 10:54:47.020939 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.020834    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-69ef21\"/\"kube-root-ca.crt\""
Sep 06 10:54:47.021026 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.020984    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-69ef21\"/\"ipp-route-69ef21-kserve-self-signed-certs\""
Sep 06 10:54:47.027060 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.027029    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:54:47.085956 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.085921    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.086123 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.085981    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.086123 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.086027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.086123 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.086081    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.086123 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.086118    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.086331 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.086240    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbm2\" (UniqueName: \"kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187240 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187204    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187240 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187242    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187483 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187363    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187483 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187592 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187507    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbm2\" (UniqueName: \"kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187592 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187540    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187659    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187780    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.187802 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.187795    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.189575 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.189540    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.189820 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.189802    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.194687 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.194658    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbm2\" (UniqueName: \"kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2\") pod \"ipp-route-69ef21-kserve-d78dbb6ff-b5r9g\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") " pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.328047 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.327960    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:47.603799 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:47.603720    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:54:47.606732 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:54:47.606710    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e90bd5a_bf6c_4e1c_bdde_9f8358c4ebf6.slice/crio-e823e6737e61f3df1fff89a4a32ae74bcf7803dc60557cbe5cdcd8a56b280af8 WatchSource:0}: Error finding container e823e6737e61f3df1fff89a4a32ae74bcf7803dc60557cbe5cdcd8a56b280af8: Status 404 returned error can't find the container with id e823e6737e61f3df1fff89a4a32ae74bcf7803dc60557cbe5cdcd8a56b280af8
Sep 06 10:54:48.509441 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:48.509406    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" event={"ID":"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6","Type":"ContainerStarted","Data":"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"}
Sep 06 10:54:48.509441 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:48.509446    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" event={"ID":"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6","Type":"ContainerStarted","Data":"e823e6737e61f3df1fff89a4a32ae74bcf7803dc60557cbe5cdcd8a56b280af8"}
Sep 06 10:54:48.524197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:48.523979    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" podStartSLOduration=2.523961444 podStartE2EDuration="2.523961444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:54:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:54:48.523398494 +0000 UTC m=+1471.693248364" watchObservedRunningTime="2026-09-06 10:54:48.523961444 +0000 UTC m=+1471.693811318"
Sep 06 10:54:57.328367 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:57.328336    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:57.328367 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:57.328377    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:57.340701 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:57.340674    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:54:57.564780 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:54:57.564752    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:55:11.828411 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:11.828381    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh_cceb09ff-9da4-45ee-a62c-c4c84140db3c/payload-processing/0.log"
Sep 06 10:55:16.629267 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.629229    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:16.633264 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.633243    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.635252 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.635231    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-676ab2-serving-cert\""
Sep 06 10:55:16.640525 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.640499    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:16.671510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.671474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.671662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.671544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6wxk\" (UniqueName: \"kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.671662 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.671597    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.671753 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.671665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.772983 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.772945    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.773210 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.773008    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6wxk\" (UniqueName: \"kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.773293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.773211    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.773293 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.773280    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.773410 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:55:16.773339    2573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-serving-cert: secret "maas-api-e2e-ipp-cleanup-676ab2-serving-cert" not found
Sep 06 10:55:16.773462 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:55:16.773422    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls podName:5e843a07-3c11-4ee1-9795-1959f2d34560 nodeName:}" failed. No retries permitted until 2026-09-06 10:55:17.273401702 +0000 UTC m=+1500.443251563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" (UID: "5e843a07-3c11-4ee1-9795-1959f2d34560") : secret "maas-api-e2e-ipp-cleanup-676ab2-serving-cert" not found
Sep 06 10:55:16.773910 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.773847    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.775671 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.775641    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:16.780981 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:16.780957    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6wxk\" (UniqueName: \"kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:17.278950 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:17.278919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:17.281402 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:17.281381    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:17.548205 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:17.548100    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:17.823947 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:17.823916    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:17.826465 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:55:17.826440    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e843a07_3c11_4ee1_9795_1959f2d34560.slice/crio-1d8a0a81aea01953e2b787fbdbff2a758a3e5191662b764f52b40623d7696c39 WatchSource:0}: Error finding container 1d8a0a81aea01953e2b787fbdbff2a758a3e5191662b764f52b40623d7696c39: Status 404 returned error can't find the container with id 1d8a0a81aea01953e2b787fbdbff2a758a3e5191662b764f52b40623d7696c39
Sep 06 10:55:17.827997 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:17.827977    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 10:55:18.656302 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:18.656267    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" event={"ID":"5e843a07-3c11-4ee1-9795-1959f2d34560","Type":"ContainerStarted","Data":"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"}
Sep 06 10:55:18.656494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:18.656321    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" event={"ID":"5e843a07-3c11-4ee1-9795-1959f2d34560","Type":"ContainerStarted","Data":"1d8a0a81aea01953e2b787fbdbff2a758a3e5191662b764f52b40623d7696c39"}
Sep 06 10:55:18.656494 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:18.656402    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:18.672137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:18.672092    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" podStartSLOduration=2.211880199 podStartE2EDuration="2.672075801s" totalImagesPullingTime="460.195602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:55:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:55:18.671178801 +0000 UTC m=+1501.841028668" watchObservedRunningTime="2026-09-06 10:55:18.672075801 +0000 UTC m=+1501.841925671"
Sep 06 10:55:24.666428 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:24.666400    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:47.226510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.226472    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:47.229055 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.226846    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" podUID="5e843a07-3c11-4ee1-9795-1959f2d34560" containerName="maas-api" containerID="cri-o://079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642" gracePeriod=30
Sep 06 10:55:47.484546 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.484488    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:47.563475 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.563433    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs\") pod \"5e843a07-3c11-4ee1-9795-1959f2d34560\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") "
Sep 06 10:55:47.563661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.563492    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") pod \"5e843a07-3c11-4ee1-9795-1959f2d34560\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") "
Sep 06 10:55:47.563661 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.563629    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6wxk\" (UniqueName: \"kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk\") pod \"5e843a07-3c11-4ee1-9795-1959f2d34560\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") "
Sep 06 10:55:47.563775 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.563688    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca\") pod \"5e843a07-3c11-4ee1-9795-1959f2d34560\" (UID: \"5e843a07-3c11-4ee1-9795-1959f2d34560\") "
Sep 06 10:55:47.564127 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.564094    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e843a07-3c11-4ee1-9795-1959f2d34560" (UID: "5e843a07-3c11-4ee1-9795-1959f2d34560"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:55:47.565838 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.565815    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e843a07-3c11-4ee1-9795-1959f2d34560" (UID: "5e843a07-3c11-4ee1-9795-1959f2d34560"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:55:47.566269 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.566247    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e843a07-3c11-4ee1-9795-1959f2d34560" (UID: "5e843a07-3c11-4ee1-9795-1959f2d34560"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:55:47.566379 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.566350    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk" (OuterVolumeSpecName: "kube-api-access-v6wxk") pod "5e843a07-3c11-4ee1-9795-1959f2d34560" (UID: "5e843a07-3c11-4ee1-9795-1959f2d34560"). InnerVolumeSpecName "kube-api-access-v6wxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:55:47.665064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.665032    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6wxk\" (UniqueName: \"kubernetes.io/projected/5e843a07-3c11-4ee1-9795-1959f2d34560-kube-api-access-v6wxk\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:55:47.665064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.665058    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e843a07-3c11-4ee1-9795-1959f2d34560-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:55:47.665064 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.665068    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:55:47.665322 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.665077    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e843a07-3c11-4ee1-9795-1959f2d34560-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:55:47.792621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.792533    2573 generic.go:356] "Generic (PLEG): container finished" podID="5e843a07-3c11-4ee1-9795-1959f2d34560" containerID="079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642" exitCode=0
Sep 06 10:55:47.792621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.792579    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"
Sep 06 10:55:47.792621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.792586    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" event={"ID":"5e843a07-3c11-4ee1-9795-1959f2d34560","Type":"ContainerDied","Data":"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"}
Sep 06 10:55:47.792887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.792671    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf" event={"ID":"5e843a07-3c11-4ee1-9795-1959f2d34560","Type":"ContainerDied","Data":"1d8a0a81aea01953e2b787fbdbff2a758a3e5191662b764f52b40623d7696c39"}
Sep 06 10:55:47.792887 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.792699    2573 scope.go:117] "RemoveContainer" containerID="079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"
Sep 06 10:55:47.801961 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.801946    2573 scope.go:117] "RemoveContainer" containerID="079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"
Sep 06 10:55:47.802237 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:55:47.802218    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642\": container with ID starting with 079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642 not found: ID does not exist" containerID="079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"
Sep 06 10:55:47.802291 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.802245    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642"} err="failed to get container status \"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642\": rpc error: code = NotFound desc = could not find container \"079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642\": container with ID starting with 079f8413b52638446e1a99a3278dfde53082c95b50878863d4f8e76ed5e7c642 not found: ID does not exist"
Sep 06 10:55:47.823573 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.823549    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:47.833697 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:47.833672    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-676ab2-5c497ffb9d-l4chf"]
Sep 06 10:55:49.352114 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:55:49.352084    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e843a07-3c11-4ee1-9795-1959f2d34560" path="/var/lib/kubelet/pods/5e843a07-3c11-4ee1-9795-1959f2d34560/volumes"
Sep 06 10:56:12.703338 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:12.703305    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:56:12.703816 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:12.703530    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" podUID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" containerName="payload-processing" containerID="cri-o://24a8dc2e613bb1057ad6446d8a25b4aa6fb6f57544d7dacc6d468a0fcf55816e" gracePeriod=30
Sep 06 10:56:12.912242 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:12.912213    2573 generic.go:356] "Generic (PLEG): container finished" podID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" containerID="24a8dc2e613bb1057ad6446d8a25b4aa6fb6f57544d7dacc6d468a0fcf55816e" exitCode=0
Sep 06 10:56:12.912389 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:12.912290    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" event={"ID":"cceb09ff-9da4-45ee-a62c-c4c84140db3c","Type":"ContainerDied","Data":"24a8dc2e613bb1057ad6446d8a25b4aa6fb6f57544d7dacc6d468a0fcf55816e"}
Sep 06 10:56:12.966705 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:12.966650    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:56:13.106715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.106681    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd7b7\" (UniqueName: \"kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7\") pod \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") "
Sep 06 10:56:13.106888 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.106731    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume\") pod \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\" (UID: \"cceb09ff-9da4-45ee-a62c-c4c84140db3c\") "
Sep 06 10:56:13.107123 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.107093    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cceb09ff-9da4-45ee-a62c-c4c84140db3c" (UID: "cceb09ff-9da4-45ee-a62c-c4c84140db3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:56:13.108877 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.108858    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7" (OuterVolumeSpecName: "kube-api-access-sd7b7") pod "cceb09ff-9da4-45ee-a62c-c4c84140db3c" (UID: "cceb09ff-9da4-45ee-a62c-c4c84140db3c"). InnerVolumeSpecName "kube-api-access-sd7b7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:56:13.207852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.207818    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7b7\" (UniqueName: \"kubernetes.io/projected/cceb09ff-9da4-45ee-a62c-c4c84140db3c-kube-api-access-sd7b7\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:13.207852 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.207845    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cceb09ff-9da4-45ee-a62c-c4c84140db3c-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:13.918226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.918189    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh" event={"ID":"cceb09ff-9da4-45ee-a62c-c4c84140db3c","Type":"ContainerDied","Data":"41629864c02401ada2ce4323c927a688d1dc370ab4dd5be8a9ea37a7ec25d7b3"}
Sep 06 10:56:13.918689 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.918238    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"
Sep 06 10:56:13.918689 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.918278    2573 scope.go:117] "RemoveContainer" containerID="24a8dc2e613bb1057ad6446d8a25b4aa6fb6f57544d7dacc6d468a0fcf55816e"
Sep 06 10:56:13.932833 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.932810    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:56:13.937098 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:13.937075    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-69ef21-bfff87f47-jf6rh"]
Sep 06 10:56:15.354056 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:15.354022    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" path="/var/lib/kubelet/pods/cceb09ff-9da4-45ee-a62c-c4c84140db3c/volumes"
Sep 06 10:56:18.216772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:18.216740    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:56:33.236685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.236621    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" podUID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" containerName="main" containerID="cri-o://ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8" gracePeriod=45
Sep 06 10:56:33.481915 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.481890    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:56:33.599585 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599476    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.599585 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599583    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wbm2\" (UniqueName: \"kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.599823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599638    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.599823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599655    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.599823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599677    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.599823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599718    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache\") pod \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\" (UID: \"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6\") "
Sep 06 10:56:33.600028 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599859    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:33.600028 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599965    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home" (OuterVolumeSpecName: "home") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:33.600028 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.599996    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:33.600376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.600095    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:33.600376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.600108    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:33.600376 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.600116    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:33.601819 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.601799    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:56:33.602196 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.602173    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm" (OuterVolumeSpecName: "dshm") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 10:56:33.602257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.602196    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2" (OuterVolumeSpecName: "kube-api-access-8wbm2") pod "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" (UID: "3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6"). InnerVolumeSpecName "kube-api-access-8wbm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:56:33.701093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.701059    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:33.701093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.701090    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbm2\" (UniqueName: \"kubernetes.io/projected/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-kube-api-access-8wbm2\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:33.701093 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:33.701099    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:56:34.013574 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.013539    2573 generic.go:356] "Generic (PLEG): container finished" podID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" containerID="ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8" exitCode=0
Sep 06 10:56:34.013795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.013580    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"
Sep 06 10:56:34.013795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.013604    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" event={"ID":"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6","Type":"ContainerDied","Data":"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"}
Sep 06 10:56:34.013795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.013645    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g" event={"ID":"3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6","Type":"ContainerDied","Data":"e823e6737e61f3df1fff89a4a32ae74bcf7803dc60557cbe5cdcd8a56b280af8"}
Sep 06 10:56:34.013795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.013668    2573 scope.go:117] "RemoveContainer" containerID="ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"
Sep 06 10:56:34.022685 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.022665    2573 scope.go:117] "RemoveContainer" containerID="ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"
Sep 06 10:56:34.022944 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:56:34.022926    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8\": container with ID starting with ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8 not found: ID does not exist" containerID="ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"
Sep 06 10:56:34.023009 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.022952    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8"} err="failed to get container status \"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8\": rpc error: code = NotFound desc = could not find container \"ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8\": container with ID starting with ff2e7aeca823a54495506b729bd4c8256024a534eabdab529c16f019ba598cb8 not found: ID does not exist"
Sep 06 10:56:34.033192 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.033148    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:56:34.036938 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:34.036916    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-69ef21/ipp-route-69ef21-kserve-d78dbb6ff-b5r9g"]
Sep 06 10:56:35.352454 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:56:35.352421    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" path="/var/lib/kubelet/pods/3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6/volumes"
Sep 06 10:57:34.638493 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.638460    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:57:34.638932 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.638674    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" podUID="79674871-ab0d-44d4-ab42-30a6487742f7" containerName="maas-api" containerID="cri-o://555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794" gracePeriod=30
Sep 06 10:57:34.688247 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.688214    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:57:34.688451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.688425    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" podUID="b539c40d-3a50-44f8-a8be-f21e123c8aca" containerName="payload-pre-processing" containerID="cri-o://0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e" gracePeriod=30
Sep 06 10:57:34.927558 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.927526    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:57:34.956366 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.956344    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:57:34.973489 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.973459    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg45g\" (UniqueName: \"kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g\") pod \"79674871-ab0d-44d4-ab42-30a6487742f7\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") "
Sep 06 10:57:34.973666 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.973578    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls\") pod \"79674871-ab0d-44d4-ab42-30a6487742f7\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") "
Sep 06 10:57:34.973666 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.973618    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca\") pod \"79674871-ab0d-44d4-ab42-30a6487742f7\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") "
Sep 06 10:57:34.973787 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.973667    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs\") pod \"79674871-ab0d-44d4-ab42-30a6487742f7\" (UID: \"79674871-ab0d-44d4-ab42-30a6487742f7\") "
Sep 06 10:57:34.974890 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.974415    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "79674871-ab0d-44d4-ab42-30a6487742f7" (UID: "79674871-ab0d-44d4-ab42-30a6487742f7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:57:34.976943 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.976915    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g" (OuterVolumeSpecName: "kube-api-access-tg45g") pod "79674871-ab0d-44d4-ab42-30a6487742f7" (UID: "79674871-ab0d-44d4-ab42-30a6487742f7"). InnerVolumeSpecName "kube-api-access-tg45g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:57:34.977412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.977390    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "79674871-ab0d-44d4-ab42-30a6487742f7" (UID: "79674871-ab0d-44d4-ab42-30a6487742f7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:57:34.977999 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:34.977972    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "79674871-ab0d-44d4-ab42-30a6487742f7" (UID: "79674871-ab0d-44d4-ab42-30a6487742f7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:57:35.075084 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075041    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbqzr\" (UniqueName: \"kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr\") pod \"b539c40d-3a50-44f8-a8be-f21e123c8aca\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") "
Sep 06 10:57:35.075303 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075126    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume\") pod \"b539c40d-3a50-44f8-a8be-f21e123c8aca\" (UID: \"b539c40d-3a50-44f8-a8be-f21e123c8aca\") "
Sep 06 10:57:35.075303 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075290    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.075303 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075300    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg45g\" (UniqueName: \"kubernetes.io/projected/79674871-ab0d-44d4-ab42-30a6487742f7-kube-api-access-tg45g\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.075457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075310    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/79674871-ab0d-44d4-ab42-30a6487742f7-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.075457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075318    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/79674871-ab0d-44d4-ab42-30a6487742f7-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.075520 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.075465    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b539c40d-3a50-44f8-a8be-f21e123c8aca" (UID: "b539c40d-3a50-44f8-a8be-f21e123c8aca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:57:35.077215 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.077196    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr" (OuterVolumeSpecName: "kube-api-access-cbqzr") pod "b539c40d-3a50-44f8-a8be-f21e123c8aca" (UID: "b539c40d-3a50-44f8-a8be-f21e123c8aca"). InnerVolumeSpecName "kube-api-access-cbqzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:57:35.176018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.175943    2573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b539c40d-3a50-44f8-a8be-f21e123c8aca-plugins-config-volume\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.176018 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.175971    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbqzr\" (UniqueName: \"kubernetes.io/projected/b539c40d-3a50-44f8-a8be-f21e123c8aca-kube-api-access-cbqzr\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:35.300031 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.299994    2573 generic.go:356] "Generic (PLEG): container finished" podID="79674871-ab0d-44d4-ab42-30a6487742f7" containerID="555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794" exitCode=0
Sep 06 10:57:35.300241 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.300040    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"
Sep 06 10:57:35.300241 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.300085    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" event={"ID":"79674871-ab0d-44d4-ab42-30a6487742f7","Type":"ContainerDied","Data":"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"}
Sep 06 10:57:35.300241 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.300142    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl" event={"ID":"79674871-ab0d-44d4-ab42-30a6487742f7","Type":"ContainerDied","Data":"0119ad3ef8f009ccf8d0bb2c8d4f0d22edeebe128042d69cd6790bd1ef2433d8"}
Sep 06 10:57:35.300241 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.300196    2573 scope.go:117] "RemoveContainer" containerID="555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"
Sep 06 10:57:35.301510 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.301488    2573 generic.go:356] "Generic (PLEG): container finished" podID="b539c40d-3a50-44f8-a8be-f21e123c8aca" containerID="0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e" exitCode=0
Sep 06 10:57:35.301601 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.301524    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"
Sep 06 10:57:35.301601 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.301521    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" event={"ID":"b539c40d-3a50-44f8-a8be-f21e123c8aca","Type":"ContainerDied","Data":"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"}
Sep 06 10:57:35.301683 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.301633    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw" event={"ID":"b539c40d-3a50-44f8-a8be-f21e123c8aca","Type":"ContainerDied","Data":"b55b2894580934f76a514513f2159801d3e0e12449f960e07f34315f9d2901de"}
Sep 06 10:57:35.309458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.309444    2573 scope.go:117] "RemoveContainer" containerID="555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"
Sep 06 10:57:35.309779 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:57:35.309757    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794\": container with ID starting with 555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794 not found: ID does not exist" containerID="555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"
Sep 06 10:57:35.309880 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.309786    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794"} err="failed to get container status \"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794\": rpc error: code = NotFound desc = could not find container \"555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794\": container with ID starting with 555ad26f76f199dbbc23c1e29fc78e51c3999f73170fe81dd476fd3651a17794 not found: ID does not exist"
Sep 06 10:57:35.309880 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.309808    2573 scope.go:117] "RemoveContainer" containerID="0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"
Sep 06 10:57:35.317953 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.317938    2573 scope.go:117] "RemoveContainer" containerID="0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"
Sep 06 10:57:35.318232 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:57:35.318208    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e\": container with ID starting with 0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e not found: ID does not exist" containerID="0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"
Sep 06 10:57:35.318289 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.318241    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e"} err="failed to get container status \"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e\": rpc error: code = NotFound desc = could not find container \"0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e\": container with ID starting with 0c1bf4083eaf474fdd402ff3aaed89123a909cdc5fde70d0acd6a23d5160709e not found: ID does not exist"
Sep 06 10:57:35.321412 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.321391    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:57:35.325635 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.325614    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2eb161-8666d5cb57-k9dzl"]
Sep 06 10:57:35.336803 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.336548    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:57:35.343356 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.343327    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2eb161-7d4f44d46-7jthw"]
Sep 06 10:57:35.352772 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.352747    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79674871-ab0d-44d4-ab42-30a6487742f7" path="/var/lib/kubelet/pods/79674871-ab0d-44d4-ab42-30a6487742f7/volumes"
Sep 06 10:57:35.353088 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:35.353076    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b539c40d-3a50-44f8-a8be-f21e123c8aca" path="/var/lib/kubelet/pods/b539c40d-3a50-44f8-a8be-f21e123c8aca/volumes"
Sep 06 10:57:52.208112 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.208081    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:57:52.208729 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.208358    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" podUID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" containerName="maas-api" containerID="cri-o://b172bc1ec2bef8015eccba4338f275d742b18f5a49ba405311ec325b5f5b95a4" gracePeriod=30
Sep 06 10:57:52.385622 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.385584    2573 generic.go:356] "Generic (PLEG): container finished" podID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" containerID="b172bc1ec2bef8015eccba4338f275d742b18f5a49ba405311ec325b5f5b95a4" exitCode=0
Sep 06 10:57:52.385823 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.385642    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" event={"ID":"6c9c64c2-6051-41bd-9e23-66c0f23f102f","Type":"ContainerDied","Data":"b172bc1ec2bef8015eccba4338f275d742b18f5a49ba405311ec325b5f5b95a4"}
Sep 06 10:57:52.456603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.456571    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:57:52.546277 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.546157    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") pod \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") "
Sep 06 10:57:52.546277 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.546249    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5jk8\" (UniqueName: \"kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8\") pod \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") "
Sep 06 10:57:52.546460 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.546299    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs\") pod \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") "
Sep 06 10:57:52.546460 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.546345    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca\") pod \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\" (UID: \"6c9c64c2-6051-41bd-9e23-66c0f23f102f\") "
Sep 06 10:57:52.546761 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.546732    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6c9c64c2-6051-41bd-9e23-66c0f23f102f" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 10:57:52.548479 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.548449    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6c9c64c2-6051-41bd-9e23-66c0f23f102f" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:57:52.548565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.548472    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8" (OuterVolumeSpecName: "kube-api-access-t5jk8") pod "6c9c64c2-6051-41bd-9e23-66c0f23f102f" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f"). InnerVolumeSpecName "kube-api-access-t5jk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:57:52.548565 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.548509    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6c9c64c2-6051-41bd-9e23-66c0f23f102f" (UID: "6c9c64c2-6051-41bd-9e23-66c0f23f102f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:57:52.647586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.647542    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:52.647586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.647580    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5jk8\" (UniqueName: \"kubernetes.io/projected/6c9c64c2-6051-41bd-9e23-66c0f23f102f-kube-api-access-t5jk8\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:52.647586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.647590    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c9c64c2-6051-41bd-9e23-66c0f23f102f-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:52.647586 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:52.647598    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c9c64c2-6051-41bd-9e23-66c0f23f102f-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:57:53.392454 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:53.392422    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"
Sep 06 10:57:53.392454 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:53.392438    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj" event={"ID":"6c9c64c2-6051-41bd-9e23-66c0f23f102f","Type":"ContainerDied","Data":"3241ea8e2396cc4b4d2f5be44971daf4a3a4bf6228e387a599726b66ceb67577"}
Sep 06 10:57:53.392954 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:53.392504    2573 scope.go:117] "RemoveContainer" containerID="b172bc1ec2bef8015eccba4338f275d742b18f5a49ba405311ec325b5f5b95a4"
Sep 06 10:57:53.406517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:53.406489    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:57:53.410503 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:53.410478    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-eb4b45-7f4bd79bc9-fndlj"]
Sep 06 10:57:55.357190 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:57:55.355499    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" path="/var/lib/kubelet/pods/6c9c64c2-6051-41bd-9e23-66c0f23f102f/volumes"
Sep 06 10:58:01.492829 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.492793    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:58:01.493451 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.493038    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" podUID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" containerName="manager" containerID="cri-o://3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf" gracePeriod=30
Sep 06 10:58:01.728583 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.728559    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:58:01.822034 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.821950    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs\") pod \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") "
Sep 06 10:58:01.822034 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.821998    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbsc9\" (UniqueName: \"kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9\") pod \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") "
Sep 06 10:58:01.822034 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.822035    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert\") pod \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\" (UID: \"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e\") "
Sep 06 10:58:01.824149 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.824120    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" (UID: "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:58:01.824303 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.824215    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9" (OuterVolumeSpecName: "kube-api-access-zbsc9") pod "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" (UID: "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e"). InnerVolumeSpecName "kube-api-access-zbsc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:58:01.824361 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.824302    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" (UID: "fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:58:01.922795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.922754    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:58:01.922795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.922790    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbsc9\" (UniqueName: \"kubernetes.io/projected/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-kube-api-access-zbsc9\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:58:01.922795 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:01.922805    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e-webhook-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:58:02.432261 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.432225    2573 generic.go:356] "Generic (PLEG): container finished" podID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" containerID="3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf" exitCode=0
Sep 06 10:58:02.432457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.432286    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd"
Sep 06 10:58:02.432457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.432298    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" event={"ID":"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e","Type":"ContainerDied","Data":"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"}
Sep 06 10:58:02.432457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.432345    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-qmzcd" event={"ID":"fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e","Type":"ContainerDied","Data":"99a1f573f8e0d218548b67064e008368465cea7f2bf2ba1bd7cd9fbd163fb22f"}
Sep 06 10:58:02.432457 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.432371    2573 scope.go:117] "RemoveContainer" containerID="3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"
Sep 06 10:58:02.441715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.441697    2573 scope.go:117] "RemoveContainer" containerID="3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"
Sep 06 10:58:02.441984 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:58:02.441965    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf\": container with ID starting with 3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf not found: ID does not exist" containerID="3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"
Sep 06 10:58:02.442031 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.441993    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf"} err="failed to get container status \"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf\": rpc error: code = NotFound desc = could not find container \"3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf\": container with ID starting with 3e990baf07176861aebe8d688a6a4e10f9efa5ff8bcd60d9b8ad02cf3f7bfeaf not found: ID does not exist"
Sep 06 10:58:02.451074 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.451051    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:58:02.453688 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:02.453668    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-qmzcd"]
Sep 06 10:58:03.353092 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:58:03.353056    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" path="/var/lib/kubelet/pods/fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e/volumes"
Sep 06 10:59:03.263150 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263116    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263487    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b539c40d-3a50-44f8-a8be-f21e123c8aca" containerName="payload-pre-processing"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263499    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b539c40d-3a50-44f8-a8be-f21e123c8aca" containerName="payload-pre-processing"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263511    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e843a07-3c11-4ee1-9795-1959f2d34560" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263517    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e843a07-3c11-4ee1-9795-1959f2d34560" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263529    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" containerName="payload-processing"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263534    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" containerName="payload-processing"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263540    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79674871-ab0d-44d4-ab42-30a6487742f7" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263545    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="79674871-ab0d-44d4-ab42-30a6487742f7" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263552    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" containerName="manager"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263557    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" containerName="manager"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263562    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263567    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" containerName="maas-api"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263575    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" containerName="main"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263580    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" containerName="main"
Sep 06 10:59:03.263629 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263631    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa7cbc33-4eb8-4a54-9ad8-ceb40e791b5e" containerName="manager"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263641    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c9c64c2-6051-41bd-9e23-66c0f23f102f" containerName="maas-api"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263648    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="79674871-ab0d-44d4-ab42-30a6487742f7" containerName="maas-api"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263653    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b539c40d-3a50-44f8-a8be-f21e123c8aca" containerName="payload-pre-processing"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263658    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="cceb09ff-9da4-45ee-a62c-c4c84140db3c" containerName="payload-processing"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263663    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e843a07-3c11-4ee1-9795-1959f2d34560" containerName="maas-api"
Sep 06 10:59:03.264226 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.263668    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e90bd5a-bf6c-4e1c-bdde-9f8358c4ebf6" containerName="main"
Sep 06 10:59:03.266714 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.266700    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.269250 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.269222    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:59:03.269250 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.269243    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h29zj\""
Sep 06 10:59:03.269458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.269224    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:59:03.269458 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.269224    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:59:03.276082 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.276054    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:03.370957 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.370922    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.371137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.371032    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rln4\" (UniqueName: \"kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.371137 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.371057    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.471943 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.471893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rln4\" (UniqueName: \"kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.471943 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.471955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.472197 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.472005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.474625 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.474584    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.474754 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.474602    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.478667 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.478648    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rln4\" (UniqueName: \"kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4\") pod \"maas-controller-5fb95fb9bd-9zlb5\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") " pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.577991 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.577883    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:03.845257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:03.845219    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:03.847638 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:59:03.847607    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3391ad2_091a_47ea_8617_46e3930676e4.slice/crio-5513e7c4a8b194b786ee652395a86e9a89ffb2fd78e25bcd86cbaf0f8383915d WatchSource:0}: Error finding container 5513e7c4a8b194b786ee652395a86e9a89ffb2fd78e25bcd86cbaf0f8383915d: Status 404 returned error can't find the container with id 5513e7c4a8b194b786ee652395a86e9a89ffb2fd78e25bcd86cbaf0f8383915d
Sep 06 10:59:04.708674 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:04.708641    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" event={"ID":"b3391ad2-091a-47ea-8617-46e3930676e4","Type":"ContainerStarted","Data":"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"}
Sep 06 10:59:04.709046 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:04.708693    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" event={"ID":"b3391ad2-091a-47ea-8617-46e3930676e4","Type":"ContainerStarted","Data":"5513e7c4a8b194b786ee652395a86e9a89ffb2fd78e25bcd86cbaf0f8383915d"}
Sep 06 10:59:04.709046 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:04.708716    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:04.723517 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:04.723474    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" podStartSLOduration=1.298446513 podStartE2EDuration="1.72345854s" totalImagesPullingTime="425.012027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:59:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:59:04.722632462 +0000 UTC m=+1727.892482348" watchObservedRunningTime="2026-09-06 10:59:04.72345854 +0000 UTC m=+1727.893308441"
Sep 06 10:59:15.715743 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:15.715713    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:25.020531 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.020497    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:25.020941 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.020685    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" podUID="b3391ad2-091a-47ea-8617-46e3930676e4" containerName="manager" containerID="cri-o://7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86" gracePeriod=30
Sep 06 10:59:25.266603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.266578    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:25.366339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.366264    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rln4\" (UniqueName: \"kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4\") pod \"b3391ad2-091a-47ea-8617-46e3930676e4\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") "
Sep 06 10:59:25.366339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.366293    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs\") pod \"b3391ad2-091a-47ea-8617-46e3930676e4\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") "
Sep 06 10:59:25.366339 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.366316    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert\") pod \"b3391ad2-091a-47ea-8617-46e3930676e4\" (UID: \"b3391ad2-091a-47ea-8617-46e3930676e4\") "
Sep 06 10:59:25.368507 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.368476    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4" (OuterVolumeSpecName: "kube-api-access-8rln4") pod "b3391ad2-091a-47ea-8617-46e3930676e4" (UID: "b3391ad2-091a-47ea-8617-46e3930676e4"). InnerVolumeSpecName "kube-api-access-8rln4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 10:59:25.368621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.368477    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b3391ad2-091a-47ea-8617-46e3930676e4" (UID: "b3391ad2-091a-47ea-8617-46e3930676e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:59:25.368621 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.368512    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b3391ad2-091a-47ea-8617-46e3930676e4" (UID: "b3391ad2-091a-47ea-8617-46e3930676e4"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 10:59:25.467105 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.467069    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rln4\" (UniqueName: \"kubernetes.io/projected/b3391ad2-091a-47ea-8617-46e3930676e4-kube-api-access-8rln4\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:59:25.467105 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.467098    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:59:25.467105 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.467106    2573 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b3391ad2-091a-47ea-8617-46e3930676e4-webhook-cert\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 10:59:25.807425 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.807392    2573 generic.go:356] "Generic (PLEG): container finished" podID="b3391ad2-091a-47ea-8617-46e3930676e4" containerID="7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86" exitCode=0
Sep 06 10:59:25.807603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.807434    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5"
Sep 06 10:59:25.807603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.807434    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" event={"ID":"b3391ad2-091a-47ea-8617-46e3930676e4","Type":"ContainerDied","Data":"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"}
Sep 06 10:59:25.807603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.807558    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-9zlb5" event={"ID":"b3391ad2-091a-47ea-8617-46e3930676e4","Type":"ContainerDied","Data":"5513e7c4a8b194b786ee652395a86e9a89ffb2fd78e25bcd86cbaf0f8383915d"}
Sep 06 10:59:25.807603 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.807585    2573 scope.go:117] "RemoveContainer" containerID="7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"
Sep 06 10:59:25.823257 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.823238    2573 scope.go:117] "RemoveContainer" containerID="7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"
Sep 06 10:59:25.823541 ip-10-0-130-119 kubenswrapper[2573]: E0906 10:59:25.823522    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86\": container with ID starting with 7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86 not found: ID does not exist" containerID="7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"
Sep 06 10:59:25.823580 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.823551    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86"} err="failed to get container status \"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86\": rpc error: code = NotFound desc = could not find container \"7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86\": container with ID starting with 7682bbc56380ad33c5590c37e1b92cac3763d02b055bef217a8b362680536d86 not found: ID does not exist"
Sep 06 10:59:25.830498 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.830471    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:25.836105 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:25.836081    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-9zlb5"]
Sep 06 10:59:26.594369 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.594333    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-nk8fk"]
Sep 06 10:59:26.594748 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.594669    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3391ad2-091a-47ea-8617-46e3930676e4" containerName="manager"
Sep 06 10:59:26.594748 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.594680    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3391ad2-091a-47ea-8617-46e3930676e4" containerName="manager"
Sep 06 10:59:26.594748 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.594749    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3391ad2-091a-47ea-8617-46e3930676e4" containerName="manager"
Sep 06 10:59:26.598948 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.598929    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.601493 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.601472    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 06 10:59:26.601600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.601540    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 06 10:59:26.601600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.601472    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 06 10:59:26.601600 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.601472    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-h29zj\""
Sep 06 10:59:26.607754 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.607732    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-nk8fk"]
Sep 06 10:59:26.677505 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.677475    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvjk\" (UniqueName: \"kubernetes.io/projected/7d064af6-c931-4c31-8b57-bcefc37419c5-kube-api-access-zvvjk\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.677715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.677544    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-webhook-cert\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.677715 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.677655    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-metrics-certs\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.778243 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.778209    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-metrics-certs\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.778243 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.778250    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvjk\" (UniqueName: \"kubernetes.io/projected/7d064af6-c931-4c31-8b57-bcefc37419c5-kube-api-access-zvvjk\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.778508 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.778284    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-webhook-cert\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.780844 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.780820    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-webhook-cert\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.780977 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.780956    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d064af6-c931-4c31-8b57-bcefc37419c5-metrics-certs\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.785867 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.785843    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvjk\" (UniqueName: \"kubernetes.io/projected/7d064af6-c931-4c31-8b57-bcefc37419c5-kube-api-access-zvvjk\") pod \"maas-controller-5fb95fb9bd-nk8fk\" (UID: \"7d064af6-c931-4c31-8b57-bcefc37419c5\") " pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:26.910324 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:26.910291    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:27.182746 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.182723    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb95fb9bd-nk8fk"]
Sep 06 10:59:27.185214 ip-10-0-130-119 kubenswrapper[2573]: W0906 10:59:27.185184    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d064af6_c931_4c31_8b57_bcefc37419c5.slice/crio-34253286aa7c08f072ad325999a36905700bb12819cc2db69ce369b49f05edad WatchSource:0}: Error finding container 34253286aa7c08f072ad325999a36905700bb12819cc2db69ce369b49f05edad: Status 404 returned error can't find the container with id 34253286aa7c08f072ad325999a36905700bb12819cc2db69ce369b49f05edad
Sep 06 10:59:27.352692 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.352658    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3391ad2-091a-47ea-8617-46e3930676e4" path="/var/lib/kubelet/pods/b3391ad2-091a-47ea-8617-46e3930676e4/volumes"
Sep 06 10:59:27.818279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.818182    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk" event={"ID":"7d064af6-c931-4c31-8b57-bcefc37419c5","Type":"ContainerStarted","Data":"81eff21371816ec6843da6b546f573311bc10e756400872f47909f5d411e63b1"}
Sep 06 10:59:27.818279 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.818238    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk" event={"ID":"7d064af6-c931-4c31-8b57-bcefc37419c5","Type":"ContainerStarted","Data":"34253286aa7c08f072ad325999a36905700bb12819cc2db69ce369b49f05edad"}
Sep 06 10:59:27.818865 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.818354    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 10:59:27.833813 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:27.833759    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk" podStartSLOduration=1.475931141 podStartE2EDuration="1.833742226s" totalImagesPullingTime="357.811085ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 10:59:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 10:59:27.832053546 +0000 UTC m=+1751.001903418" watchObservedRunningTime="2026-09-06 10:59:27.833742226 +0000 UTC m=+1751.003592097"
Sep 06 10:59:38.825411 ip-10-0-130-119 kubenswrapper[2573]: I0906 10:59:38.825379    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb95fb9bd-nk8fk"
Sep 06 11:02:17.420191 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.420136    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:17.424265 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.424241    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.426213 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.426189    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6eb51a07-kserve-self-signed-certs\""
Sep 06 11:02:17.432966 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.432937    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:17.477508 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.477699 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477525    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.477699 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477577    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9zq\" (UniqueName: \"kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.477699 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477648    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.477892 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477751    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.477892 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.477812    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578379 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578592 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578395    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578592 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578426    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578592 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578459    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578592 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578592 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578520    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9zq\" (UniqueName: \"kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578871 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578817    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.578924 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.578864    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.579054 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.579032    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.580939 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.580909    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.581321 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.581295    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.585392 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.585369    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9zq\" (UniqueName: \"kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq\") pod \"e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") " pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:17.735762 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:17.735656    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:18.006217 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:18.006117    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:18.010217 ip-10-0-130-119 kubenswrapper[2573]: W0906 11:02:18.010190    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4d27e50_a983_40d8_8b55_8d9d8036d9df.slice/crio-10388e74310629160af6ed26687872c956456aafd22bc3b4f684dc700cb89367 WatchSource:0}: Error finding container 10388e74310629160af6ed26687872c956456aafd22bc3b4f684dc700cb89367: Status 404 returned error can't find the container with id 10388e74310629160af6ed26687872c956456aafd22bc3b4f684dc700cb89367
Sep 06 11:02:18.588549 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:18.588509    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" event={"ID":"b4d27e50-a983-40d8-8b55-8d9d8036d9df","Type":"ContainerStarted","Data":"fa5a8decbded102e1142e2e6600cc2fce6b3f39186d153982de1a6ae24906dbe"}
Sep 06 11:02:18.588940 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:18.588565    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" event={"ID":"b4d27e50-a983-40d8-8b55-8d9d8036d9df","Type":"ContainerStarted","Data":"10388e74310629160af6ed26687872c956456aafd22bc3b4f684dc700cb89367"}
Sep 06 11:02:18.603907 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:18.603854    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" podStartSLOduration=1.603837849 podStartE2EDuration="1.603837849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 11:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 11:02:18.602207981 +0000 UTC m=+1921.772057852" watchObservedRunningTime="2026-09-06 11:02:18.603837849 +0000 UTC m=+1921.773687720"
Sep 06 11:02:27.735833 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:27.735793    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:27.736284 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:27.735872    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:27.748513 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:27.748489    2573 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:28.647362 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:28.647329    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:37.512667 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:37.512624    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:52.535681 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.535632    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" podUID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" containerName="main" containerID="cri-o://fa5a8decbded102e1142e2e6600cc2fce6b3f39186d153982de1a6ae24906dbe" gracePeriod=45
Sep 06 11:02:52.745754 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.745722    2573 generic.go:356] "Generic (PLEG): container finished" podID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" containerID="fa5a8decbded102e1142e2e6600cc2fce6b3f39186d153982de1a6ae24906dbe" exitCode=0
Sep 06 11:02:52.745916 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.745793    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" event={"ID":"b4d27e50-a983-40d8-8b55-8d9d8036d9df","Type":"ContainerDied","Data":"fa5a8decbded102e1142e2e6600cc2fce6b3f39186d153982de1a6ae24906dbe"}
Sep 06 11:02:52.780818 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.780791    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:52.917571 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917529    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917768 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917579    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917768 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917620    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917768 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917646    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917768 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917749    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf9zq\" (UniqueName: \"kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917976 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917781    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm\") pod \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\" (UID: \"b4d27e50-a983-40d8-8b55-8d9d8036d9df\") "
Sep 06 11:02:52.917976 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917936    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home" (OuterVolumeSpecName: "home") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 11:02:52.918064 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.917921    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 11:02:52.918131 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.918110    2573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-model-cache\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:52.918217 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.918135    2573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-home\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:52.918259 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.918237    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 11:02:52.919897 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.919866    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 11:02:52.920010 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.919915    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm" (OuterVolumeSpecName: "dshm") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 06 11:02:52.920134 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:52.920112    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq" (OuterVolumeSpecName: "kube-api-access-nf9zq") pod "b4d27e50-a983-40d8-8b55-8d9d8036d9df" (UID: "b4d27e50-a983-40d8-8b55-8d9d8036d9df"). InnerVolumeSpecName "kube-api-access-nf9zq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 11:02:53.019318 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.019277    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf9zq\" (UniqueName: \"kubernetes.io/projected/b4d27e50-a983-40d8-8b55-8d9d8036d9df-kube-api-access-nf9zq\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:53.019318 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.019311    2573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-dshm\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:53.019318 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.019326    2573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tls-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:53.019556 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.019338    2573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4d27e50-a983-40d8-8b55-8d9d8036d9df-tmp-dir\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:02:53.751643 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.751607    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"
Sep 06 11:02:53.751643 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.751617    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz" event={"ID":"b4d27e50-a983-40d8-8b55-8d9d8036d9df","Type":"ContainerDied","Data":"10388e74310629160af6ed26687872c956456aafd22bc3b4f684dc700cb89367"}
Sep 06 11:02:53.752154 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.751672    2573 scope.go:117] "RemoveContainer" containerID="fa5a8decbded102e1142e2e6600cc2fce6b3f39186d153982de1a6ae24906dbe"
Sep 06 11:02:53.766721 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.766697    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:53.770121 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:53.770098    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6eb51a07-kserve-6477bd45f5-vpspz"]
Sep 06 11:02:55.352235 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:02:55.352199    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" path="/var/lib/kubelet/pods/b4d27e50-a983-40d8-8b55-8d9d8036d9df/volumes"
Sep 06 11:03:02.560489 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.560443    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"]
Sep 06 11:03:02.561103 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.560832    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" containerName="main"
Sep 06 11:03:02.561103 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.560844    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" containerName="main"
Sep 06 11:03:02.561103 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.560918    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4d27e50-a983-40d8-8b55-8d9d8036d9df" containerName="main"
Sep 06 11:03:02.565285 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.565258    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.573884 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.573381    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"]
Sep 06 11:03:02.602099 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.602054    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-openshift-service-ca\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.602099 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.602099    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmjtw\" (UniqueName: \"kubernetes.io/projected/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-kube-api-access-vmjtw\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.602356 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.602132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-metrics-certs\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.602356 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.602206    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-maas-api-tls\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.702941 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.702903    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-openshift-service-ca\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.702941 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.702939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmjtw\" (UniqueName: \"kubernetes.io/projected/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-kube-api-access-vmjtw\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.703213 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.702984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-metrics-certs\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.703213 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.703064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-maas-api-tls\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.703769 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.703745    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-openshift-service-ca\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.705897 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.705874    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-maas-api-tls\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.706010 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.705921    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-metrics-certs\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.710252 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.710221    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmjtw\" (UniqueName: \"kubernetes.io/projected/f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6-kube-api-access-vmjtw\") pod \"maas-api-66b9899c65-6d2zz\" (UID: \"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6\") " pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:02.878254 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:02.878135    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:03.152515 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.152478    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"]
Sep 06 11:03:03.155812 ip-10-0-130-119 kubenswrapper[2573]: W0906 11:03:03.155779    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16f24a4_57ed_4a5c_b1c0_30a9b26d53f6.slice/crio-eb91c2ce83fc3dc89cc9b0e9fc8242031d7f7ef027451505eb3de50f2ef7210b WatchSource:0}: Error finding container eb91c2ce83fc3dc89cc9b0e9fc8242031d7f7ef027451505eb3de50f2ef7210b: Status 404 returned error can't find the container with id eb91c2ce83fc3dc89cc9b0e9fc8242031d7f7ef027451505eb3de50f2ef7210b
Sep 06 11:03:03.157354 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.157324    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 06 11:03:03.800634 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.800599    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz" event={"ID":"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6","Type":"ContainerStarted","Data":"cdebc723454d784223c78a648d34c54ca5c72ba68e6a89d65773a1ab64d36029"}
Sep 06 11:03:03.801044 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.800658    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz" event={"ID":"f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6","Type":"ContainerStarted","Data":"eb91c2ce83fc3dc89cc9b0e9fc8242031d7f7ef027451505eb3de50f2ef7210b"}
Sep 06 11:03:03.801044 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.800684    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:03.815738 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:03.815679    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz" podStartSLOduration=1.294012411 podStartE2EDuration="1.815656316s" totalImagesPullingTime="521.643905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 11:03:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 11:03:03.81455456 +0000 UTC m=+1966.984404431" watchObservedRunningTime="2026-09-06 11:03:03.815656316 +0000 UTC m=+1966.985506192"
Sep 06 11:03:09.810613 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:09.810578    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66b9899c65-6d2zz"
Sep 06 11:03:17.160798 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:17.160761    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 11:03:17.161316 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:17.161054    2573 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" podUID="ffad3759-5647-49bb-a1aa-9d82d9380be6" containerName="maas-api" containerID="cri-o://f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e" gracePeriod=30
Sep 06 11:03:18.410721 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.410693    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 11:03:18.448274 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.448235    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgscd\" (UniqueName: \"kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd\") pod \"ffad3759-5647-49bb-a1aa-9d82d9380be6\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") "
Sep 06 11:03:18.448456 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.448296    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls\") pod \"ffad3759-5647-49bb-a1aa-9d82d9380be6\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") "
Sep 06 11:03:18.448521 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.448495    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca\") pod \"ffad3759-5647-49bb-a1aa-9d82d9380be6\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") "
Sep 06 11:03:18.448609 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.448592    2573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs\") pod \"ffad3759-5647-49bb-a1aa-9d82d9380be6\" (UID: \"ffad3759-5647-49bb-a1aa-9d82d9380be6\") "
Sep 06 11:03:18.448880 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.448852    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ffad3759-5647-49bb-a1aa-9d82d9380be6" (UID: "ffad3759-5647-49bb-a1aa-9d82d9380be6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 06 11:03:18.450616 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.450583    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ffad3759-5647-49bb-a1aa-9d82d9380be6" (UID: "ffad3759-5647-49bb-a1aa-9d82d9380be6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 11:03:18.450763 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.450676    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd" (OuterVolumeSpecName: "kube-api-access-fgscd") pod "ffad3759-5647-49bb-a1aa-9d82d9380be6" (UID: "ffad3759-5647-49bb-a1aa-9d82d9380be6"). InnerVolumeSpecName "kube-api-access-fgscd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 06 11:03:18.450956 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.450927    2573 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ffad3759-5647-49bb-a1aa-9d82d9380be6" (UID: "ffad3759-5647-49bb-a1aa-9d82d9380be6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 06 11:03:18.550079 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.549995    2573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-maas-api-tls\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:03:18.550079 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.550024    2573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffad3759-5647-49bb-a1aa-9d82d9380be6-openshift-service-ca\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:03:18.550079 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.550033    2573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffad3759-5647-49bb-a1aa-9d82d9380be6-metrics-certs\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:03:18.550079 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.550042    2573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgscd\" (UniqueName: \"kubernetes.io/projected/ffad3759-5647-49bb-a1aa-9d82d9380be6-kube-api-access-fgscd\") on node \"ip-10-0-130-119.ec2.internal\" DevicePath \"\""
Sep 06 11:03:18.873747 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.873660    2573 generic.go:356] "Generic (PLEG): container finished" podID="ffad3759-5647-49bb-a1aa-9d82d9380be6" containerID="f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e" exitCode=0
Sep 06 11:03:18.873747 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.873707    2573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"
Sep 06 11:03:18.873747 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.873724    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" event={"ID":"ffad3759-5647-49bb-a1aa-9d82d9380be6","Type":"ContainerDied","Data":"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"}
Sep 06 11:03:18.874006 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.873767    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr" event={"ID":"ffad3759-5647-49bb-a1aa-9d82d9380be6","Type":"ContainerDied","Data":"3564cb2955f5d31dae2792ec82088590b217eb1169e22ebcf2c0030c53cc812a"}
Sep 06 11:03:18.874006 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.873786    2573 scope.go:117] "RemoveContainer" containerID="f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"
Sep 06 11:03:18.883030 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.883012    2573 scope.go:117] "RemoveContainer" containerID="f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"
Sep 06 11:03:18.883331 ip-10-0-130-119 kubenswrapper[2573]: E0906 11:03:18.883309    2573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e\": container with ID starting with f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e not found: ID does not exist" containerID="f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"
Sep 06 11:03:18.883412 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.883342    2573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e"} err="failed to get container status \"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e\": rpc error: code = NotFound desc = could not find container \"f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e\": container with ID starting with f542d546fad4b5fa188edd1fc09e782d819560ccb58dad94aa37e7840a132e2e not found: ID does not exist"
Sep 06 11:03:18.893103 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.893074    2573 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 11:03:18.896646 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:18.896621    2573 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-bd6c55749-b4hvr"]
Sep 06 11:03:19.352330 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:19.352294    2573 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffad3759-5647-49bb-a1aa-9d82d9380be6" path="/var/lib/kubelet/pods/ffad3759-5647-49bb-a1aa-9d82d9380be6/volumes"
Sep 06 11:03:37.538908 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:37.538867    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754d676c85-js5nj_448c56bc-285c-4bf7-aaf9-7577e72f8503/authorino/0.log"
Sep 06 11:03:42.791927 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:42.791895    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-d85fq_3ecd2b97-0377-4bc4-a1ca-2c290354a8ee/manager/0.log"
Sep 06 11:03:42.911827 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:42.911795    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb95fb9bd-nk8fk_7d064af6-c931-4c31-8b57-bcefc37419c5/manager/0.log"
Sep 06 11:03:43.504075 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:43.504042    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645f664568-z9k79_a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9/copy-manifests/0.log"
Sep 06 11:03:43.510590 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:43.510548    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645f664568-z9k79_a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9/manager/0.log"
Sep 06 11:03:43.843855 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:43.843775    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-66b9899c65-6d2zz_f16f24a4-57ed-4a5c-b1c0-30a9b26d53f6/maas-api/0.log"
Sep 06 11:03:44.273464 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:44.273376    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-mr7vk_a67ed0b9-27a8-4dba-9697-56dc26b3308f/postgres/0.log"
Sep 06 11:03:45.018053 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:45.018016    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754d676c85-js5nj_448c56bc-285c-4bf7-aaf9-7577e72f8503/authorino/0.log"
Sep 06 11:03:45.361678 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:45.361599    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-69wl2_7939deb1-d286-4728-a093-bedcc4be0510/kuadrant-console-plugin/0.log"
Sep 06 11:03:46.414745 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:46.414712    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586db6db5-wpksw_516a8da8-1848-4b52-a302-e3d06e03d8f7/kube-auth-proxy/0.log"
Sep 06 11:03:46.647260 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:46.647221    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-b4xt6_3a9c6cc5-aa86-4c1e-9417-f68aaf204a71/payload-pre-processing/0.log"
Sep 06 11:03:47.339547 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:47.339519    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-dtqzv_ae440408-3afe-4a78-9822-15145e540b8e/main/0.log"
Sep 06 11:03:47.659190 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:47.659153    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frgsl7_19158881-0979-4ec5-97d4-0f433497dca5/main/0.log"
Sep 06 11:03:53.163559 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.163517    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pw9j6/must-gather-s2n9k"]
Sep 06 11:03:53.164070 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.164047    2573 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffad3759-5647-49bb-a1aa-9d82d9380be6" containerName="maas-api"
Sep 06 11:03:53.164146 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.164075    2573 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffad3759-5647-49bb-a1aa-9d82d9380be6" containerName="maas-api"
Sep 06 11:03:53.164240 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.164185    2573 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffad3759-5647-49bb-a1aa-9d82d9380be6" containerName="maas-api"
Sep 06 11:03:53.167463 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.167438    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.169728 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.169706    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pw9j6\"/\"kube-root-ca.crt\""
Sep 06 11:03:53.169851 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.169749    2573 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pw9j6\"/\"openshift-service-ca.crt\""
Sep 06 11:03:53.170317 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.170300    2573 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pw9j6\"/\"default-dockercfg-xq2z4\""
Sep 06 11:03:53.175343 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.175317    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pw9j6/must-gather-s2n9k"]
Sep 06 11:03:53.263419 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.263380    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2n5c\" (UniqueName: \"kubernetes.io/projected/65a570c1-4b69-4833-a16a-1f0fd7b68d59-kube-api-access-b2n5c\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.263419 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.263422    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65a570c1-4b69-4833-a16a-1f0fd7b68d59-must-gather-output\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.363961 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.363921    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2n5c\" (UniqueName: \"kubernetes.io/projected/65a570c1-4b69-4833-a16a-1f0fd7b68d59-kube-api-access-b2n5c\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.364186 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.363966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65a570c1-4b69-4833-a16a-1f0fd7b68d59-must-gather-output\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.364347 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.364328    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/65a570c1-4b69-4833-a16a-1f0fd7b68d59-must-gather-output\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.371492 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.371473    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2n5c\" (UniqueName: \"kubernetes.io/projected/65a570c1-4b69-4833-a16a-1f0fd7b68d59-kube-api-access-b2n5c\") pod \"must-gather-s2n9k\" (UID: \"65a570c1-4b69-4833-a16a-1f0fd7b68d59\") " pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.477184 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.477086    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pw9j6/must-gather-s2n9k"
Sep 06 11:03:53.751846 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:53.751800    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pw9j6/must-gather-s2n9k"]
Sep 06 11:03:53.754407 ip-10-0-130-119 kubenswrapper[2573]: W0906 11:03:53.754380    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65a570c1_4b69_4833_a16a_1f0fd7b68d59.slice/crio-684a75c4bf89e8e4b86a4834e72c6cb8445d016cfa2aa2d03168eead0953da9b WatchSource:0}: Error finding container 684a75c4bf89e8e4b86a4834e72c6cb8445d016cfa2aa2d03168eead0953da9b: Status 404 returned error can't find the container with id 684a75c4bf89e8e4b86a4834e72c6cb8445d016cfa2aa2d03168eead0953da9b
Sep 06 11:03:54.043348 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:54.043257    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pw9j6/must-gather-s2n9k" event={"ID":"65a570c1-4b69-4833-a16a-1f0fd7b68d59","Type":"ContainerStarted","Data":"684a75c4bf89e8e4b86a4834e72c6cb8445d016cfa2aa2d03168eead0953da9b"}
Sep 06 11:03:55.051915 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:55.051874    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pw9j6/must-gather-s2n9k" event={"ID":"65a570c1-4b69-4833-a16a-1f0fd7b68d59","Type":"ContainerStarted","Data":"d290af772b6068fb1d32b77dc442371a325bcc18b941a31729073a5686708b56"}
Sep 06 11:03:55.052339 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:55.051935    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pw9j6/must-gather-s2n9k" event={"ID":"65a570c1-4b69-4833-a16a-1f0fd7b68d59","Type":"ContainerStarted","Data":"d753be573be58462d5a8750018f9836c27d019c14cd63323d6b55e88fff818ab"}
Sep 06 11:03:55.066540 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:55.066487    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pw9j6/must-gather-s2n9k" podStartSLOduration=1.339892923 podStartE2EDuration="2.066469022s" totalImagesPullingTime="726.576099ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-06 11:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 11:03:55.065293081 +0000 UTC m=+2018.235142955" watchObservedRunningTime="2026-09-06 11:03:55.066469022 +0000 UTC m=+2018.236318893"
Sep 06 11:03:56.048979 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:56.048950    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-n764h_7508edae-7d98-4088-a13f-ac16080cdf40/global-pull-secret-syncer/0.log"
Sep 06 11:03:56.159201 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:56.159156    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wsb9h_281a9014-bf0e-409e-8d9f-42982ba2595e/konnectivity-agent/0.log"
Sep 06 11:03:56.181142 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:03:56.181113    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-119.ec2.internal_d150e533eb692c08b346a2806d019c7d/haproxy/0.log"
Sep 06 11:04:01.476777 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:01.476746    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-754d676c85-js5nj_448c56bc-285c-4bf7-aaf9-7577e72f8503/authorino/0.log"
Sep 06 11:04:01.553956 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:01.553919    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-69wl2_7939deb1-d286-4728-a093-bedcc4be0510/kuadrant-console-plugin/0.log"
Sep 06 11:04:03.361830 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.361781    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-n2n2q_261b99cc-dc72-43d8-aa15-b3040591c16e/kube-state-metrics/0.log"
Sep 06 11:04:03.381293 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.381211    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-n2n2q_261b99cc-dc72-43d8-aa15-b3040591c16e/kube-rbac-proxy-main/0.log"
Sep 06 11:04:03.403104 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.403069    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-n2n2q_261b99cc-dc72-43d8-aa15-b3040591c16e/kube-rbac-proxy-self/0.log"
Sep 06 11:04:03.438530 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.438449    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-99d9f67d-f79h6_e5013522-89bb-4749-a539-a0b4263c1a73/metrics-server/0.log"
Sep 06 11:04:03.465092 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.465069    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-q2spn_abefd3c4-dc1e-41aa-b694-755fc8163d18/monitoring-plugin/0.log"
Sep 06 11:04:03.606645 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.606589    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2ddf_cf1f9b21-a9c3-4068-a795-0f4114b70ced/node-exporter/0.log"
Sep 06 11:04:03.635596 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.635519    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2ddf_cf1f9b21-a9c3-4068-a795-0f4114b70ced/kube-rbac-proxy/0.log"
Sep 06 11:04:03.660484 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.660460    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2ddf_cf1f9b21-a9c3-4068-a795-0f4114b70ced/init-textfile/0.log"
Sep 06 11:04:03.845637 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.845609    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/prometheus/0.log"
Sep 06 11:04:03.871190 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.871143    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/config-reloader/0.log"
Sep 06 11:04:03.892121 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.892079    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/thanos-sidecar/0.log"
Sep 06 11:04:03.916631 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.916598    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/kube-rbac-proxy-web/0.log"
Sep 06 11:04:03.940465 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.940419    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/kube-rbac-proxy/0.log"
Sep 06 11:04:03.967049 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.967018    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/kube-rbac-proxy-thanos/0.log"
Sep 06 11:04:03.994616 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:03.994589    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_8b2a8237-b7c3-4a1f-aac0-f9ff3d4dbff4/init-config-reloader/0.log"
Sep 06 11:04:04.044597 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:04.044548    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-dx74m_4ee43b66-46fd-4432-9d0d-fb7cfaf5243a/prometheus-operator/0.log"
Sep 06 11:04:04.068656 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:04.068628    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-dx74m_4ee43b66-46fd-4432-9d0d-fb7cfaf5243a/kube-rbac-proxy/0.log"
Sep 06 11:04:04.137088 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:04.137055    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55b7fcfdcf-6dcfd_7898c289-f50a-4cf7-8121-238838f172c9/telemeter-client/0.log"
Sep 06 11:04:04.157110 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:04.157023    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55b7fcfdcf-6dcfd_7898c289-f50a-4cf7-8121-238838f172c9/reload/0.log"
Sep 06 11:04:04.178276 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:04.178241    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55b7fcfdcf-6dcfd_7898c289-f50a-4cf7-8121-238838f172c9/kube-rbac-proxy/0.log"
Sep 06 11:04:06.558209 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.558147    2573 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"]
Sep 06 11:04:06.566420 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.566389    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.568268 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.568234    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"]
Sep 06 11:04:06.711983 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.711938    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-podres\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.712203 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.712004    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrk5t\" (UniqueName: \"kubernetes.io/projected/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-kube-api-access-wrk5t\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.712203 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.712045    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-proc\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.712203 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.712103    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-lib-modules\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.712203 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.712143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-sys\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.812947 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.812854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-podres\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813126 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.812947    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrk5t\" (UniqueName: \"kubernetes.io/projected/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-kube-api-access-wrk5t\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813126 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.812988    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-proc\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813126 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813041    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-lib-modules\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813126 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813079    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-sys\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813126 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813092    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-podres\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813431 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813218    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-proc\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813431 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813355    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-lib-modules\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.813431 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.813414    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-sys\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.820855 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.820823    2573 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrk5t\" (UniqueName: \"kubernetes.io/projected/dcd19f9a-c268-4b35-ae8f-0c5c12536c3c-kube-api-access-wrk5t\") pod \"perf-node-gather-daemonset-jzrsg\" (UID: \"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c\") " pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:06.873320 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.873283    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-rs548_ffcbe0a7-c7aa-49f1-974a-290a0701164b/download-server/0.log"
Sep 06 11:04:06.880050 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:06.880019    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:07.189372 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:07.189330    2573 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"]
Sep 06 11:04:07.191862 ip-10-0-130-119 kubenswrapper[2573]: W0906 11:04:07.191815    2573 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddcd19f9a_c268_4b35_ae8f_0c5c12536c3c.slice/crio-1faf4bd76d2b3931ca3053d48f4645086814263131dce2a61f3cf54a287b6ff7 WatchSource:0}: Error finding container 1faf4bd76d2b3931ca3053d48f4645086814263131dce2a61f3cf54a287b6ff7: Status 404 returned error can't find the container with id 1faf4bd76d2b3931ca3053d48f4645086814263131dce2a61f3cf54a287b6ff7
Sep 06 11:04:08.144110 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.144066    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg" event={"ID":"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c","Type":"ContainerStarted","Data":"2d05454014835e2d011511cbcee2bdb43ec2ac576a8259685d2e4d80c05bf9d1"}
Sep 06 11:04:08.144603 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.144134    2573 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg" event={"ID":"dcd19f9a-c268-4b35-ae8f-0c5c12536c3c","Type":"ContainerStarted","Data":"1faf4bd76d2b3931ca3053d48f4645086814263131dce2a61f3cf54a287b6ff7"}
Sep 06 11:04:08.144603 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.144384    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:08.158542 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.158476    2573 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg" podStartSLOduration=2.158453605 podStartE2EDuration="2.158453605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-06 11:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-06 11:04:08.157710771 +0000 UTC m=+2031.327560644" watchObservedRunningTime="2026-09-06 11:04:08.158453605 +0000 UTC m=+2031.328303478"
Sep 06 11:04:08.775222 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.775193    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9whf_e2a06dd0-f982-4bea-a575-147f2ea38801/dns/0.log"
Sep 06 11:04:08.794864 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.794831    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-x9whf_e2a06dd0-f982-4bea-a575-147f2ea38801/kube-rbac-proxy/0.log"
Sep 06 11:04:08.817423 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:08.817395    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hc7gh_b95d18fd-4d41-449d-89e8-dc637b17d5bd/dns-node-resolver/0.log"
Sep 06 11:04:09.453406 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:09.453378    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rzvjc_81643241-0445-4393-9b5c-028981d2dfe0/node-ca/0.log"
Sep 06 11:04:10.962021 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:10.961982    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586db6db5-wpksw_516a8da8-1848-4b52-a302-e3d06e03d8f7/kube-auth-proxy/0.log"
Sep 06 11:04:11.037979 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:11.037939    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-b4xt6_3a9c6cc5-aa86-4c1e-9417-f68aaf204a71/payload-pre-processing/0.log"
Sep 06 11:04:11.975257 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:11.975221    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bhl5s_98dcb88b-0924-45a3-870b-8fae24b02936/serve-healthcheck-canary/0.log"
Sep 06 11:04:12.552046 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:12.552006    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5fvv_408d6e0d-907d-42d3-9364-9b2e37e3ed6d/kube-rbac-proxy/0.log"
Sep 06 11:04:12.571210 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:12.571152    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5fvv_408d6e0d-907d-42d3-9364-9b2e37e3ed6d/exporter/0.log"
Sep 06 11:04:12.592015 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:12.591967    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n5fvv_408d6e0d-907d-42d3-9364-9b2e37e3ed6d/extractor/0.log"
Sep 06 11:04:14.160057 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:14.160018    2573 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pw9j6/perf-node-gather-daemonset-jzrsg"
Sep 06 11:04:15.037349 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:15.037285    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-d85fq_3ecd2b97-0377-4bc4-a1ca-2c290354a8ee/manager/0.log"
Sep 06 11:04:15.103948 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:15.102922    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb95fb9bd-nk8fk_7d064af6-c931-4c31-8b57-bcefc37419c5/manager/0.log"
Sep 06 11:04:15.459270 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:15.459239    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645f664568-z9k79_a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9/manager/0.log"
Sep 06 11:04:15.479286 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:15.479260    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-645f664568-z9k79_a4a719c3-ee83-406e-a3d1-4f69a8eb9cc9/copy-manifests/0.log"
Sep 06 11:04:16.919835 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:16.919808    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-59b7bf7857-pzkbz_2a0e893c-665f-490d-abfe-c3de0f5cee53/manager/0.log"
Sep 06 11:04:17.771760 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:17.771730    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586db6db5-wpksw_516a8da8-1848-4b52-a302-e3d06e03d8f7/kube-auth-proxy/0.log"
Sep 06 11:04:17.842286 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:17.842230    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-b4xt6_3a9c6cc5-aa86-4c1e-9417-f68aaf204a71/payload-pre-processing/0.log"
Sep 06 11:04:19.945289 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:19.945260    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5586db6db5-wpksw_516a8da8-1848-4b52-a302-e3d06e03d8f7/kube-auth-proxy/0.log"
Sep 06 11:04:20.010832 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:20.010794    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-b4xt6_3a9c6cc5-aa86-4c1e-9417-f68aaf204a71/payload-pre-processing/0.log"
Sep 06 11:04:23.275911 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.275872    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6zhpb_971fbef9-6745-4123-88a4-fc2c9649b359/kube-multus/0.log"
Sep 06 11:04:23.433355 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.433326    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/kube-multus-additional-cni-plugins/0.log"
Sep 06 11:04:23.453442 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.453413    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/egress-router-binary-copy/0.log"
Sep 06 11:04:23.472711 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.472689    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/cni-plugins/0.log"
Sep 06 11:04:23.491508 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.491481    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/bond-cni-plugin/0.log"
Sep 06 11:04:23.509801 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.509770    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/routeoverride-cni/0.log"
Sep 06 11:04:23.529013 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.528935    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/whereabouts-cni-bincopy/0.log"
Sep 06 11:04:23.547508 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.547479    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kzp48_631ae72d-88b6-4777-8e72-53906c337bb2/whereabouts-cni/0.log"
Sep 06 11:04:23.762206 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.762140    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dkvt2_4e2f82a0-2b33-4585-b5a4-072fadb0f89d/network-metrics-daemon/0.log"
Sep 06 11:04:23.777900 ip-10-0-130-119 kubenswrapper[2573]: I0906 11:04:23.777875    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-dkvt2_4e2f82a0-2b33-4585-b5a4-072fadb0f89d/kube-rbac-proxy/0.log"