Sep 12 01:45:04.863787 ip-10-0-128-81 systemd[1]: Starting Kubernetes Kubelet...
Sep 12 01:45:05.286292 ip-10-0-128-81 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 01:45:05.286292 ip-10-0-128-81 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 12 01:45:05.286292 ip-10-0-128-81 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 01:45:05.286292 ip-10-0-128-81 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 12 01:45:05.286292 ip-10-0-128-81 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 12 01:45:05.287503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.287421    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 12 01:45:05.291785 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291769    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.291785 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291784    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291788    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291791    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291795    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291798    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291804    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291808    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291811    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291814    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291817    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291821    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291824    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291826    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291829    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291832    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291835    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291837    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291840    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291842    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.291846 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291845    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291847    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291850    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291861    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291864    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291866    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291869    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291871    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291873    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291876    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291878    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291881    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291883    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291886    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291888    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291890    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291893    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291895    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291898    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.292302 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291900    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291902    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291905    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291908    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291910    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291913    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291915    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291917    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291920    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291923    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291925    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291927    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291930    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291932    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291936    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291940    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291944    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291947    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291950    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.292811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291953    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291955    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291958    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291961    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291963    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291967    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291969    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291972    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291974    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291977    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291979    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291983    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291986    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291990    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291993    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291995    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.291998    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292000    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292003    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292005    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.293276 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292008    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292010    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292014    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292016    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292019    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292021    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292024    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292026    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292031    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292034    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292036    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292039    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292042    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292044    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292047    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292049    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292052    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292055    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292057    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292060    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.293771 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.292063    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293581    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293590    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293593    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293596    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293598    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293602    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293606    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293608    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293611    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293613    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293616    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293619    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293621    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293624    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293626    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293629    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293632    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293634    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293637    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.294256 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293640    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293643    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293661    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293666    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293669    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293672    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293675    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293677    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293680    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293682    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293685    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293687    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293689    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293692    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293695    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293697    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293700    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293704    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293708    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293711    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.294761 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293714    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293717    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293719    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293722    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293725    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293727    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293730    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293733    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293735    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293738    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293741    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293743    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293746    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293749    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293752    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293755    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293757    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293759    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293762    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293764    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.295291 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293767    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293770    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293773    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293776    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293780    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293783    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293786    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293789    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293792    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293794    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293797    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293799    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293802    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293805    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293807    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293810    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293813    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293815    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293818    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.295801 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293821    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293823    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293826    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293828    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293831    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293834    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293836    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293839    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293841    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293844    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293846    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293849    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293852    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293854    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293857    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293860    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293864    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293867    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293870    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.296262 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293873    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.293877    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294227    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294232    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294235    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294237    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294241    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294244    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294246    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294249    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294251    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294254    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294257    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294259    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294262    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294267    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294270    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294273    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294277    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294279    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.296735 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294282    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294284    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294287    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294290    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294293    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294296    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294298    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294301    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294303    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294306    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294308    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294311    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294313    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294315    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294318    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294321    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294323    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294327    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294330    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.297210 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294334    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294337    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294339    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294342    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294345    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294348    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294350    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294353    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294355    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294358    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294360    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294363    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294366    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294368    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294371    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294373    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294376    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294378    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294381    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294384    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.297693 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294386    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294389    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294392    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294394    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294396    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294399    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294401    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294404    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294406    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294408    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294411    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294413    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294416    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294418    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294421    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294423    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294426    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294428    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294431    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294433    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.298171 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294436    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294439    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294442    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294446    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294449    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294451    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294453    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294456    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294458    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294461    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294463    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294466    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294468    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294471    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294473    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294476    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294478    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294481    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294483    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.298672 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294486    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294488    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.294490    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294578    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294629    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294640    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294649    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294655    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294660    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294664    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294668    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294671    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294674    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294678    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294681    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294684    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294687    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294690    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294693    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294695    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294698    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294702    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294705    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294708    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 12 01:45:05.299145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294711    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294714    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294718    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294720    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294724    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294727    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294730    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294732    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294735    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294738    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294741    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294745    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294748    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294751    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294753    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294756    2582 flags.go:64] FLAG: --enable-server="true"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294760    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294765    2582 flags.go:64] FLAG: --event-burst="100"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294768    2582 flags.go:64] FLAG: --event-qps="50"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294771    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294774    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294777    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294781    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294786    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294789    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 12 01:45:05.299721 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294792    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294796    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294799    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294802    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294804    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294807    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294810    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294813    2582 flags.go:64] FLAG: --feature-gates=""
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294817    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294820    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294822    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294826    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294829    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294832    2582 flags.go:64] FLAG: --help="false"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294835    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294838    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294840    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294843    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294847    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294851    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294854    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294856    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294859    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 12 01:45:05.300316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294862    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294865    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294869    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294871    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294874    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294877    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294880    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294884    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294889    2582 flags.go:64] FLAG: --lock-file=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294892    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294895    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294898    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294903    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294906    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294909    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294912    2582 flags.go:64] FLAG: --logging-format="text"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294914    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294918    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294921    2582 flags.go:64] FLAG: --manifest-url=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294924    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294928    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294931    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294943    2582 flags.go:64] FLAG: --max-pods="110"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294946    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294949    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 12 01:45:05.300936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294952    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294955    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294958    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294961    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294964    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294970    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294973    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294976    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294979    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294983    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294988    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294991    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294994    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.294997    2582 flags.go:64] FLAG: --port="10250"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295001    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295004    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06519223e9a05dbc5"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295008    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295011    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295014    2582 flags.go:64] FLAG: --register-node="true"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295017    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295020    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295023    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295026    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295029    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295037    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 12 01:45:05.301548 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295041    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295044    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295046    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295049    2582 flags.go:64] FLAG: --runonce="false"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295052    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295055    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295058    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295061    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295063    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295066    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295069    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295072    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295075    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295078    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295081    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295083    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295086    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295090    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295092    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295097    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295100    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295105    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295109    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295113    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295117    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 12 01:45:05.302140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295119    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295122    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295125    2582 flags.go:64] FLAG: --v="2"
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295129    2582 flags.go:64] FLAG: --version="false"
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295133    2582 flags.go:64] FLAG: --vmodule=""
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295137    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.295140    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295248    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295254    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295257    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295260    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295263    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295266    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295268    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295272    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295275    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295278    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295280    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295283    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295286    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295289    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.302754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295291    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295294    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295297    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295301    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295305    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295308    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295311    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295316    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295318    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295322    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295325    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295327    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295330    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295333    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295335    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295338    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295342    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295345    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295349    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.303277 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295352    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295354    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295357    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295359    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295362    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295365    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295367    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295370    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295373    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295375    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295378    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295380    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295383    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295385    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295388    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295390    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295393    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295395    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295404    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295407    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.303752 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295411    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295413    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295417    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295420    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295423    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295425    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295428    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295430    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295433    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295435    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295438    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295441    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295443    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295446    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295449    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295451    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295454    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295456    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295458    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295461    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.304279 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295464    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295467    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295469    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295472    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295474    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295477    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295480    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295482    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295485    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295487    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295490    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295493    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295497    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295499    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295504    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295507    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295522    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295525    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295528    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295530    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.304774 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295533    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295535    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295538    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295541    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295543    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.295546    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.296075    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.302833    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.302848    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.302861    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.302867    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302921    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302927    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302941    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302946    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.305331 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302949    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302952    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302955    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302957    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302960    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302963    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302966    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302968    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302972    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302975    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302978    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302980    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302983    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302986    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302989    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302993    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.302996    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303000    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303003    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.305754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303005    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303008    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303011    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303014    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303017    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303019    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303023    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303026    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303029    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303031    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303034    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303036    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303039    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303041    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303044    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303047    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303049    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303052    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303055    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303057    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.306224 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303060    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303063    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303067    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303071    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303073    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303076    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303078    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303081    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303083    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303086    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303088    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303091    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303093    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303096    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303098    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303101    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303103    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303106    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303109    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.306726 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303112    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303114    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303117    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303119    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303122    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303124    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303127    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303130    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303132    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303135    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303137    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303140    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303142    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303144    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303147    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303149    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303152    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303155    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303157    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303160    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.307200 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303162    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303165    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303167    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303170    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303172    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303175    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303177    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303180    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303182    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303185    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303187    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303190    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303193    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303196    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303198    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303201    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.307722 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303204    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.303209    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303318    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303323    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303326    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303329    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303332    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303334    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303337    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303339    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303342    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303344    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303347    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303350    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303352    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 12 01:45:05.308125 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303355    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303358    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303360    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303363    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303365    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303368    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303370    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303373    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303375    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303378    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303380    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303383    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303385    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303388    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303391    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303393    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303396    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303398    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303401    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303403    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 12 01:45:05.308489 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303406    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303408    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303411    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303413    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303416    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303418    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303421    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303423    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303426    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303428    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303430    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303433    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303436    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303438    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303441    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303443    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303447    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303451    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303454    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303456    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 12 01:45:05.308998 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303459    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303461    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303464    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303466    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303469    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303472    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303474    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303477    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303479    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303482    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303484    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303487    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303489    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303492    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303496    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303499    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303501    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303504    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303507    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 12 01:45:05.309491 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303522    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303525    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303528    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303531    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303534    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303538    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303541    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303544    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303546    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303549    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303552    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303555    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303557    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303560    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303562    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303564    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303567    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303569    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303574    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 12 01:45:05.310044 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303577    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303580    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303583    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303585    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303588    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303591    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303594    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:05.303596    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.303601    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.303694    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.305579    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.306567    2582 server.go:1019] "Starting client certificate rotation"
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.306665    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 01:45:05.310596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.306706    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 12 01:45:05.311120 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.309351    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 01:45:05.313259 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.313238    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 12 01:45:05.324599 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.324582    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 12 01:45:05.328636 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.328623    2582 log.go:25] "Validated CRI v1 image API"
Sep 12 01:45:05.331464 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.331449    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 12 01:45:05.335187 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.335168    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8063621e-cd67-449a-b83e-9b373e2a6314:/dev/nvme0n1p4 879e8bfa-51bc-4583-b5bc-f09f8bfb2d45:/dev/nvme0n1p3]
Sep 12 01:45:05.335252 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.335185    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 12 01:45:05.337420 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.337404    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 01:45:05.340855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.340745    2582 manager.go:216] Machine: {Timestamp:2026-09-12 01:45:05.339145492 +0000 UTC m=+0.355616414 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3065046 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c48a90c640e145b8877b0473a3922 SystemUUID:ec2c48a9-0c64-0e14-5b88-77b0473a3922 BootID:a3a21934-425e-430a-acf7-acb4b2dd6173 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b4:2a:9f:91:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b4:2a:9f:91:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:b1:92:50:c0:2d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 12 01:45:05.340855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.340847    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 12 01:45:05.340962 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.340915    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 12 01:45:05.344000 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.343981    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 12 01:45:05.344155 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.344002    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-81.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 12 01:45:05.344788 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.344778    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 12 01:45:05.344828 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.344790    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 12 01:45:05.344828 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.344798    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 01:45:05.344828 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.344811    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 12 01:45:05.345598 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.345588    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 12 01:45:05.349083 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349072    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 01:45:05.349189 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349180    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 12 01:45:05.349219 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349211    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 12 01:45:05.349252 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349227    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 12 01:45:05.349252 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349251    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 12 01:45:05.349303 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349263    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 12 01:45:05.349303 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349278    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 12 01:45:05.349641 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349629    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 01:45:05.349681 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.349649    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 12 01:45:05.351917 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.351902    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 12 01:45:05.353768 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.353746    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 12 01:45:05.353856 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.353790    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 12 01:45:05.355310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355295    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355316    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355324    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355332    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355344    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355352    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355371    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355379    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355388    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 12 01:45:05.355402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355396    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 12 01:45:05.355685 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355408    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 12 01:45:05.355685 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.355431    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 12 01:45:05.356261 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.356250    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 12 01:45:05.356318 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.356264    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 12 01:45:05.359797 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.359782    2582 server.go:1268] "Started kubelet"
Sep 12 01:45:05.360119 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.360013    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 12 01:45:05.364417 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.364358    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fdrz"
Sep 12 01:45:05.364701 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.364684    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-81.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 12 01:45:05.364815 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.364783    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 12 01:45:05.364873 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.364848    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 12 01:45:05.365041 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.364994    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 12 01:45:05.365061 ip-10-0-128-81 systemd[1]: Started Kubernetes Kubelet.
Sep 12 01:45:05.365447 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.365423    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-81.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 12 01:45:05.366918 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.366902    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 12 01:45:05.367389 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.367178    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8fdrz"
Sep 12 01:45:05.367819 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.367799    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 12 01:45:05.368404 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.368355    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 12 01:45:05.372598 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.371554    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-81.ec2.internal.18d46f762f16870c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-81.ec2.internal,UID:ip-10-0-128-81.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-81.ec2.internal,},FirstTimestamp:2026-09-12 01:45:05.359759116 +0000 UTC m=+0.376230038,LastTimestamp:2026-09-12 01:45:05.359759116 +0000 UTC m=+0.376230038,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-81.ec2.internal,}"
Sep 12 01:45:05.372673 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.372666    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 12 01:45:05.373306 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.373293    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 12 01:45:05.373415 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.373398    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.373966 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.373933    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 12 01:45:05.374063 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.373969    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 12 01:45:05.374063 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.373981    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 12 01:45:05.374173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374086    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 12 01:45:05.374173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374098    2582 factory.go:55] Registering systemd factory
Sep 12 01:45:05.374173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374104    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 12 01:45:05.374173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374105    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 12 01:45:05.374173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374113    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 12 01:45:05.374414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374298    2582 factory.go:153] Registering CRI-O factory
Sep 12 01:45:05.374414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374306    2582 factory.go:223] Registration of the crio container factory successfully
Sep 12 01:45:05.374414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374319    2582 factory.go:103] Registering Raw factory
Sep 12 01:45:05.374414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374332    2582 manager.go:1196] Started watching for new ooms in manager
Sep 12 01:45:05.374666 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.374655    2582 manager.go:349] Starting recovery of all containers
Sep 12 01:45:05.377602 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.377584    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:05.381903 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.381882    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-81.ec2.internal\" not found" node="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.385806 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.385791    2582 manager.go:354] Recovery completed
Sep 12 01:45:05.390106 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.390093    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.391889 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.391873    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.391954 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.391910    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.391954 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.391927    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.392475 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.392460    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 12 01:45:05.393851 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.392476    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 12 01:45:05.393902 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.393861    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 12 01:45:05.393955 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.393931    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.395072 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.395055    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.395205 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.395195    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.395309 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.395290    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.397207 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.397184    2582 policy_none.go:49] "None policy: Start"
Sep 12 01:45:05.397207 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.397201    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 12 01:45:05.397341 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.397230    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 12 01:45:05.399185 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.399161    2582 policy_none.go:47] "Start"
Sep 12 01:45:05.457912 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.457776    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 12 01:45:05.460005 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.459043    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 12 01:45:05.460005 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.459091    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 12 01:45:05.460005 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.459113    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 12 01:45:05.460005 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.459178    2582 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 12 01:45:05.461417 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.461399    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:05.474377 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.474353    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.559616 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.559600    2582 kubelet.go:2472] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Sep 12 01:45:05.564047 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564037    2582 manager.go:337] "Starting Device Plugin manager"
Sep 12 01:45:05.564112 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.564069    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 12 01:45:05.564112 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564079    2582 server.go:87] "Starting device plugin registration server"
Sep 12 01:45:05.564365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564354    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 12 01:45:05.564444 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564365    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 12 01:45:05.564499 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564487    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 12 01:45:05.564587 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564573    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 12 01:45:05.564587 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.564588    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 12 01:45:05.565086 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.565068    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 12 01:45:05.565148 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.565120    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.665116 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.665088    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.665897 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.665881    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.665975 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.665907    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.665975 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.665918    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.665975 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.665942    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.673910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.673897    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.673958 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.673916    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-81.ec2.internal\": node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.687962 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.687943    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.760443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.760411    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"]
Sep 12 01:45:05.760563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.760490    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.761404 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.761391    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.761475 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.761420    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.761475 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.761432    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.763821 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.763809    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.763942 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.763927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.763995 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.763958    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.764487 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764472    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.764567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764490    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.764567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764499    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.764567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764532    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.764567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764546    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.764567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.764549    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.766735 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.766718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.766805 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.766750    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 12 01:45:05.767384 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.767371    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientMemory"
Sep 12 01:45:05.767435 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.767394    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasNoDiskPressure"
Sep 12 01:45:05.767435 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.767403    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeHasSufficientPID"
Sep 12 01:45:05.775791 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.775777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.775855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.775800    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.775855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.775817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c31d8c8119d4fdd6c9aac0791f891f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"8c31d8c8119d4fdd6c9aac0791f891f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.788825 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.788769    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.796171 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.796150    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-81.ec2.internal\" not found" node="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.800549 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.800536    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-81.ec2.internal\" not found" node="ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876214    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876307 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876237    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876307 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876254    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c31d8c8119d4fdd6c9aac0791f891f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"8c31d8c8119d4fdd6c9aac0791f891f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876371 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8c31d8c8119d4fdd6c9aac0791f891f3-config\") pod \"kube-apiserver-proxy-ip-10-0-128-81.ec2.internal\" (UID: \"8c31d8c8119d4fdd6c9aac0791f891f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876371 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876315    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.876371 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:05.876326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d6911f5f0618fc5e9567bb786e74c577-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal\" (UID: \"d6911f5f0618fc5e9567bb786e74c577\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:05.889503 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.889484    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:05.990207 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:05.990178    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:06.090747 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:06.090683    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:06.098850 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:06.098826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:06.103824 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:06.103721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:07.280792 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.280759    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.381243 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.381217    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.396384 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.396361    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 12 01:45:07.396532 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.396515    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 01:45:07.396580 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.396524    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 12 01:45:07.461829 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.461764    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-11 01:40:06 +0000 UTC" deadline="2028-04-20 11:56:51.145394144 +0000 UTC"
Sep 12 01:45:07.461829 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.461789    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14074h11m43.683608168s"
Sep 12 01:45:07.482266 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.482246    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.524256 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.524237    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:07.583257 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.583237    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.683874 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.683852    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.784307 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.784279    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.877015 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:07.876981    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6911f5f0618fc5e9567bb786e74c577.slice/crio-e43ad74bdbf773e0f5fd05fbd6d61bb599d5d90de177f5b096fc3bcaed9f628e WatchSource:0}: Error finding container e43ad74bdbf773e0f5fd05fbd6d61bb599d5d90de177f5b096fc3bcaed9f628e: Status 404 returned error can't find the container with id e43ad74bdbf773e0f5fd05fbd6d61bb599d5d90de177f5b096fc3bcaed9f628e
Sep 12 01:45:07.877572 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:07.877551    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c31d8c8119d4fdd6c9aac0791f891f3.slice/crio-441a60e3bbae9e2068858d65845c6d50f6dbf2c6cc2e3b0bc754b2d39204c547 WatchSource:0}: Error finding container 441a60e3bbae9e2068858d65845c6d50f6dbf2c6cc2e3b0bc754b2d39204c547: Status 404 returned error can't find the container with id 441a60e3bbae9e2068858d65845c6d50f6dbf2c6cc2e3b0bc754b2d39204c547
Sep 12 01:45:07.883397 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.883380    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:45:07.884986 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:07.884966    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-81.ec2.internal\" not found"
Sep 12 01:45:07.899867 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.899845    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:07.963784 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.963752    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:07.974177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.972684    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 01:45:07.975185 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.975166    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal"
Sep 12 01:45:07.984239 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:07.984216    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 12 01:45:08.357028 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.356964    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:08.440701 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.440476    2582 apiserver.go:52] "Watching apiserver"
Sep 12 01:45:08.447132 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.447104    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 12 01:45:08.447624 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.447598    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2","openshift-multus/multus-additional-cni-plugins-r2blc","openshift-multus/multus-nwflt","openshift-multus/network-metrics-daemon-xk2nm","kube-system/global-pull-secret-syncer-dzpgh","openshift-cluster-node-tuning-operator/tuned-b8dsz","openshift-dns/node-resolver-mvbd7","openshift-image-registry/node-ca-4ndgk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal","openshift-network-diagnostics/network-check-target-pb84r","openshift-network-operator/iptables-alerter-wbszw","openshift-ovn-kubernetes/ovnkube-node-2spkx","kube-system/konnectivity-agent-x5zgg"]
Sep 12 01:45:08.450314 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.450295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.453244 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.453225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6krwn\""
Sep 12 01:45:08.453901 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.453434    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.453901 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.453467    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.453901 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.453487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.455912    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456145    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456323    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456356    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456573    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bvbdj\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.456843    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 12 01:45:08.457053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.457053    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.457458 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.457297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.459165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.458879    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 12 01:45:08.459165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.458935    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 12 01:45:08.459165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.458979    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nr4ld\""
Sep 12 01:45:08.459165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.458994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.459389 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.459176    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 12 01:45:08.459389 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.459324    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.459389 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.459387    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bb27x\""
Sep 12 01:45:08.459596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.459398    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 12 01:45:08.460641 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.460623    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.460726 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.460699    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:08.462615 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.462594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.462709 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.462672    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:08.462907 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.462891    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 12 01:45:08.465004 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.464985    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.469365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.467717    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.469365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.467962    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.469365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.468209    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r7n6p\""
Sep 12 01:45:08.469365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.468418    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 12 01:45:08.469365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.468633    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.470362 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.470345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.471069 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.471023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 12 01:45:08.471350 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.471179    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cbpvq\""
Sep 12 01:45:08.472070 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.471902    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 12 01:45:08.472499 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.472481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.472839 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.472824    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 12 01:45:08.474359 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.472984    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gc2gn\""
Sep 12 01:45:08.474775 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.473099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.475207 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.473404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-netns\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-systemd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475277    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.475399 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlh87\" (UniqueName: \"kubernetes.io/projected/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-kube-api-access-wlh87\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.475399 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-kubernetes\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475399 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475399 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475376    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-sys\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-k8s-cni-cncf-io\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475449    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-bin\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475471    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-env-overrides\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-tuned\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-tmp\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-kubelet\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.475585 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7s76\" (UniqueName: \"kubernetes.io/projected/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-kube-api-access-s7s76\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475616    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-slash\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-etc-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475698    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-conf-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-dbus\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475764    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-conf\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-cni-binary-copy\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-ovn\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-node-log\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-log-socket\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.475906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-netd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-lib-modules\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.475987    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-config\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-script-lib\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd5x\" (UniqueName: \"kubernetes.io/projected/4db996df-7bb9-42a4-9c57-497e46831b9e-kube-api-access-8bd5x\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.476126    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-kubelet\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476163    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476210    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-system-cni-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-modprobe-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476264    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-daemon-config\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476300    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-multus-certs\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476323    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-var-lib-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.476556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/685ab9df-4254-4f9e-8208-30d659158759-ovn-node-metrics-cert\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysconfig\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-host\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-hostroot\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lb98\" (UniqueName: \"kubernetes.io/projected/440dbf68-d46f-4711-9c48-cbd0ede84229-kube-api-access-2lb98\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-serviceca\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-cnibin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-netns\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-kubelet-config\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476630    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkqv\" (UniqueName: \"kubernetes.io/projected/8d726d2d-af38-4e62-84f9-67bf9533132b-kube-api-access-qbkqv\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-system-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cnibin\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-host\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-run\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-bin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-multus\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477310 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476897    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmf9k\" (UniqueName: \"kubernetes.io/projected/685ab9df-4254-4f9e-8208-30d659158759-kube-api-access-tmf9k\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476919    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-systemd\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-var-lib-kubelet\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-etc-kubernetes\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476981    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.476998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.477021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-socket-dir-parent\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.477059    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-systemd-units\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.477980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.477384    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.478517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.478501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.478650 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.478633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.478850 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.478834    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wgttv\""
Sep 12 01:45:08.479245 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.479226    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 12 01:45:08.480829 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.480810    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 12 01:45:08.481158 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.480955    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 12 01:45:08.481158 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.481031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hsmvm\""
Sep 12 01:45:08.481158 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.481110    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 12 01:45:08.496789 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.496767    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtl54"
Sep 12 01:45:08.503718 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.503695    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rtl54"
Sep 12 01:45:08.558694 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.558641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"8c31d8c8119d4fdd6c9aac0791f891f3","Type":"ContainerStarted","Data":"441a60e3bbae9e2068858d65845c6d50f6dbf2c6cc2e3b0bc754b2d39204c547"}
Sep 12 01:45:08.562104 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.562077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"d6911f5f0618fc5e9567bb786e74c577","Type":"ContainerStarted","Data":"e43ad74bdbf773e0f5fd05fbd6d61bb599d5d90de177f5b096fc3bcaed9f628e"}
Sep 12 01:45:08.565700 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.564730    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 12 01:45:08.578021 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.577999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-kubernetes\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578133 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578133 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-sys\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578242 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578154    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-kubernetes\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578242 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-k8s-cni-cncf-io\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.578242 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578216    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-bin\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578242 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-env-overrides\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-tuned\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-tmp\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-kubelet\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7s76\" (UniqueName: \"kubernetes.io/projected/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-kube-api-access-s7s76\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.578434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-slash\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-etc-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-sys-fs\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-conf-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-dbus\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3ac547-fd72-474e-91ff-cc7123edffd7-agent-certs\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578580    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-iptables-alerter-script\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578621    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-conf\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578651    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-cni-binary-copy\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-ovn\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-node-log\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-log-socket\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-netd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-socket-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.578807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-lib-modules\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578888    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-config\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-script-lib\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch28j\" (UniqueName: \"kubernetes.io/projected/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-kube-api-access-ch28j\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4gzv\" (UniqueName: \"kubernetes.io/projected/498ffecf-2695-4c69-bf14-c6bf64599f72-kube-api-access-p4gzv\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.578990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd5x\" (UniqueName: \"kubernetes.io/projected/4db996df-7bb9-42a4-9c57-497e46831b9e-kube-api-access-8bd5x\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579050    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-kubelet\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579078    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579124    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-host-slash\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579205    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-conf-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579242    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-dbus\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579256    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-kubelet\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-ovn\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.579461 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579316    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-bin\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-etc-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysctl-conf\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579388    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2120641c-d0f4-41db-b3bc-734724f523b7-hosts-file\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579414    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-lib-modules\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-system-cni-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-sys\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-modprobe-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579468    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-kubelet\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-daemon-config\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-multus-certs\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-ovn-kubernetes\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-var-lib-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-slash\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-k8s-cni-cncf-io\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.579500    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:08.580278 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.579632    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:09.079591191 +0000 UTC m=+3.006537077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-multus-certs\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-node-log\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-log-socket\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.579966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-config\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-cni-netd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-cni-binary-copy\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580159    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-var-lib-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/685ab9df-4254-4f9e-8208-30d659158759-ovn-node-metrics-cert\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580221    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-modprobe-d\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysconfig\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580274    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-host\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-system-cni-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-hostroot\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-env-overrides\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lb98\" (UniqueName: \"kubernetes.io/projected/440dbf68-d46f-4711-9c48-cbd0ede84229-kube-api-access-2lb98\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120641c-d0f4-41db-b3bc-734724f523b7-tmp-dir\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-serviceca\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-cnibin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580589    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-netns\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580594    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-host\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-kubelet-config\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580621    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/685ab9df-4254-4f9e-8208-30d659158759-ovnkube-script-lib\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-daemon-config\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkqv\" (UniqueName: \"kubernetes.io/projected/8d726d2d-af38-4e62-84f9-67bf9533132b-kube-api-access-qbkqv\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-system-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580720    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cnibin\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-host\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-system-cni-dir\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580813    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-run\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.581858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-bin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580851    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7fff888-e442-4677-a3c9-9b5d490bb332-kubelet-config\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580881    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-multus\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cnibin\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580907    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580931    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-cnibin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.580932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmf9k\" (UniqueName: \"kubernetes.io/projected/685ab9df-4254-4f9e-8208-30d659158759-kube-api-access-tmf9k\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581014    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-serviceca\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581088    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-systemd\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-var-lib-kubelet\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581165    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-run-netns\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-openvswitch\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581201    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-host\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-hostroot\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-etc-kubernetes\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-multus\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581257    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-run\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.582696 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-var-lib-kubelet\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-sysconfig\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581329    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-host-var-lib-cni-bin\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-systemd\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-socket-dir-parent\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581402    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-etc-kubernetes\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/440dbf68-d46f-4711-9c48-cbd0ede84229-multus-socket-dir-parent\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-systemd-units\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581462    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-netns\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-host-run-netns\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-systemd-units\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-systemd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnsw7\" (UniqueName: \"kubernetes.io/projected/2120641c-d0f4-41db-b3bc-734724f523b7-kube-api-access-qnsw7\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/685ab9df-4254-4f9e-8208-30d659158759-run-systemd\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581690    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlh87\" (UniqueName: \"kubernetes.io/projected/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-kube-api-access-wlh87\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581805    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.581814    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:08.583525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3ac547-fd72-474e-91ff-cc7123edffd7-konnectivity-ca\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.581896    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:09.081877053 +0000 UTC m=+3.008822931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-registration-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.581956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-device-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.582006    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.582603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-tmp\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.583149    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d726d2d-af38-4e62-84f9-67bf9533132b-etc-tuned\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.584283 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.583406    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/685ab9df-4254-4f9e-8208-30d659158759-ovn-node-metrics-cert\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.588907 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.588884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkqv\" (UniqueName: \"kubernetes.io/projected/8d726d2d-af38-4e62-84f9-67bf9533132b-kube-api-access-qbkqv\") pod \"tuned-b8dsz\" (UID: \"8d726d2d-af38-4e62-84f9-67bf9533132b\") " pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.589547 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.589526    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd5x\" (UniqueName: \"kubernetes.io/projected/4db996df-7bb9-42a4-9c57-497e46831b9e-kube-api-access-8bd5x\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:08.589645 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.589620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7s76\" (UniqueName: \"kubernetes.io/projected/e2d2ce75-d3cd-41cc-8c67-70c3deeb0196-kube-api-access-s7s76\") pod \"multus-additional-cni-plugins-r2blc\" (UID: \"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196\") " pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.591285 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.591252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lb98\" (UniqueName: \"kubernetes.io/projected/440dbf68-d46f-4711-9c48-cbd0ede84229-kube-api-access-2lb98\") pod \"multus-nwflt\" (UID: \"440dbf68-d46f-4711-9c48-cbd0ede84229\") " pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.591434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.591416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmf9k\" (UniqueName: \"kubernetes.io/projected/685ab9df-4254-4f9e-8208-30d659158759-kube-api-access-tmf9k\") pod \"ovnkube-node-2spkx\" (UID: \"685ab9df-4254-4f9e-8208-30d659158759\") " pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.593578 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.593554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlh87\" (UniqueName: \"kubernetes.io/projected/0c82b21e-bb74-4edf-9d48-e8bafbdc3037-kube-api-access-wlh87\") pod \"node-ca-4ndgk\" (UID: \"0c82b21e-bb74-4edf-9d48-e8bafbdc3037\") " pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.632608 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.632552    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 12 01:45:08.682915 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.682889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-sys-fs\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683002 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.682927    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3ac547-fd72-474e-91ff-cc7123edffd7-agent-certs\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.683002 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.682953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-iptables-alerter-script\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.683002 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.682981    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-socket-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch28j\" (UniqueName: \"kubernetes.io/projected/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-kube-api-access-ch28j\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-sys-fs\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683031    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4gzv\" (UniqueName: \"kubernetes.io/projected/498ffecf-2695-4c69-bf14-c6bf64599f72-kube-api-access-p4gzv\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683101    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-host-slash\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2120641c-d0f4-41db-b3bc-734724f523b7-hosts-file\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.683190 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683209    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120641c-d0f4-41db-b3bc-734724f523b7-tmp-dir\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnsw7\" (UniqueName: \"kubernetes.io/projected/2120641c-d0f4-41db-b3bc-734724f523b7-kube-api-access-qnsw7\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3ac547-fd72-474e-91ff-cc7123edffd7-konnectivity-ca\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-registration-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-device-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:08.683866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683667    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-host-slash\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.683866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683729    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-etc-selinux\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683760    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-socket-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.683866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2120641c-d0f4-41db-b3bc-734724f523b7-hosts-file\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.683866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683823    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.684162 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-device-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.684162 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.683961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/498ffecf-2695-4c69-bf14-c6bf64599f72-registration-dir\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.684162 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.684069    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2120641c-d0f4-41db-b3bc-734724f523b7-tmp-dir\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.684337 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.684273    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-iptables-alerter-script\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.684399 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.684381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c3ac547-fd72-474e-91ff-cc7123edffd7-konnectivity-ca\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.686097 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.686079    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c3ac547-fd72-474e-91ff-cc7123edffd7-agent-certs\") pod \"konnectivity-agent-x5zgg\" (UID: \"1c3ac547-fd72-474e-91ff-cc7123edffd7\") " pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.690351 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.690332    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:08.690351 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.690354    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:08.690517 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.690369    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:08.690517 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:08.690422    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:09.190405539 +0000 UTC m=+3.117351404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:08.693738 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.693709    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnsw7\" (UniqueName: \"kubernetes.io/projected/2120641c-d0f4-41db-b3bc-734724f523b7-kube-api-access-qnsw7\") pod \"node-resolver-mvbd7\" (UID: \"2120641c-d0f4-41db-b3bc-734724f523b7\") " pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.693911 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.693884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4gzv\" (UniqueName: \"kubernetes.io/projected/498ffecf-2695-4c69-bf14-c6bf64599f72-kube-api-access-p4gzv\") pod \"aws-ebs-csi-driver-node-c5jb2\" (UID: \"498ffecf-2695-4c69-bf14-c6bf64599f72\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.695452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.695430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch28j\" (UniqueName: \"kubernetes.io/projected/5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99-kube-api-access-ch28j\") pod \"iptables-alerter-wbszw\" (UID: \"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99\") " pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:08.766064 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.765995    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-b8dsz"
Sep 12 01:45:08.778210 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.778183    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:08.787120 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.787096    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2blc"
Sep 12 01:45:08.792924 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.792892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwflt"
Sep 12 01:45:08.801195 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.801176    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4ndgk"
Sep 12 01:45:08.810845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.810825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:08.817393 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.817377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2"
Sep 12 01:45:08.825879 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.825860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mvbd7"
Sep 12 01:45:08.834403 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:08.834388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wbszw"
Sep 12 01:45:09.086599 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.086513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:09.086757 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.086602    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:09.086757 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.086684    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:09.086757 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.086733    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:09.086906 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.086776    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:10.086755165 +0000 UTC m=+4.013701243 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:09.086906 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.086799    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:10.086787342 +0000 UTC m=+4.013733208 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:09.292881 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.288596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:09.292881 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.288804    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:09.292881 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.288825    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:09.292881 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.288839    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:09.292881 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.288905    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:10.288886026 +0000 UTC m=+4.215831903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:09.507493 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.507390    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 01:40:08 +0000 UTC" deadline="2028-05-15 21:36:39.26215707 +0000 UTC"
Sep 12 01:45:09.507493 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.507439    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14683h51m29.754721534s"
Sep 12 01:45:09.549694 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:09.549658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:09.549814 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:09.549771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:09.776270 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.776246    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498ffecf_2695_4c69_bf14_c6bf64599f72.slice/crio-c5659ec76871d102b397c9e22c92bbf38df91ecccd00957c97f4972d033e78af WatchSource:0}: Error finding container c5659ec76871d102b397c9e22c92bbf38df91ecccd00957c97f4972d033e78af: Status 404 returned error can't find the container with id c5659ec76871d102b397c9e22c92bbf38df91ecccd00957c97f4972d033e78af
Sep 12 01:45:09.777811 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.777694    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb6bff6_c91f_4c5a_aeb3_86e7e61a0c99.slice/crio-1d4fac731768fc926434b9aa0fbaab71e9f626a7a928b0b7ba9a7df8d748dd95 WatchSource:0}: Error finding container 1d4fac731768fc926434b9aa0fbaab71e9f626a7a928b0b7ba9a7df8d748dd95: Status 404 returned error can't find the container with id 1d4fac731768fc926434b9aa0fbaab71e9f626a7a928b0b7ba9a7df8d748dd95
Sep 12 01:45:09.780558 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.780526    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2120641c_d0f4_41db_b3bc_734724f523b7.slice/crio-952922d3f8f51b56f127f1e4d5444323659039c85fc3dc4b6fe1c2400fbd81dd WatchSource:0}: Error finding container 952922d3f8f51b56f127f1e4d5444323659039c85fc3dc4b6fe1c2400fbd81dd: Status 404 returned error can't find the container with id 952922d3f8f51b56f127f1e4d5444323659039c85fc3dc4b6fe1c2400fbd81dd
Sep 12 01:45:09.781720 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.781688    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c82b21e_bb74_4edf_9d48_e8bafbdc3037.slice/crio-054d87dc6807e729072cf0851fb2b52e5e657dd71c46e948729a4df77aca507c WatchSource:0}: Error finding container 054d87dc6807e729072cf0851fb2b52e5e657dd71c46e948729a4df77aca507c: Status 404 returned error can't find the container with id 054d87dc6807e729072cf0851fb2b52e5e657dd71c46e948729a4df77aca507c
Sep 12 01:45:09.782499 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.782481    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440dbf68_d46f_4711_9c48_cbd0ede84229.slice/crio-99b1e06fa2866dff41f8160042994af77070625329ef3bd87e20d3a601258771 WatchSource:0}: Error finding container 99b1e06fa2866dff41f8160042994af77070625329ef3bd87e20d3a601258771: Status 404 returned error can't find the container with id 99b1e06fa2866dff41f8160042994af77070625329ef3bd87e20d3a601258771
Sep 12 01:45:09.783750 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.783719    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d726d2d_af38_4e62_84f9_67bf9533132b.slice/crio-24272a07bc248bfdc2d3b52f0763785f7b1a12570dd9c964653945ace6b86fe2 WatchSource:0}: Error finding container 24272a07bc248bfdc2d3b52f0763785f7b1a12570dd9c964653945ace6b86fe2: Status 404 returned error can't find the container with id 24272a07bc248bfdc2d3b52f0763785f7b1a12570dd9c964653945ace6b86fe2
Sep 12 01:45:09.784595 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.784569    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod685ab9df_4254_4f9e_8208_30d659158759.slice/crio-e15f6af2a0de5765a60a329c6baacd87f796b0a1d1be73eadedbc76a4e2523a7 WatchSource:0}: Error finding container e15f6af2a0de5765a60a329c6baacd87f796b0a1d1be73eadedbc76a4e2523a7: Status 404 returned error can't find the container with id e15f6af2a0de5765a60a329c6baacd87f796b0a1d1be73eadedbc76a4e2523a7
Sep 12 01:45:09.786670 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.786648    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d2ce75_d3cd_41cc_8c67_70c3deeb0196.slice/crio-4a4a713b1766482f295903c3152f793bc41ea551da15a2ae58fe5f43a7930e96 WatchSource:0}: Error finding container 4a4a713b1766482f295903c3152f793bc41ea551da15a2ae58fe5f43a7930e96: Status 404 returned error can't find the container with id 4a4a713b1766482f295903c3152f793bc41ea551da15a2ae58fe5f43a7930e96
Sep 12 01:45:09.786843 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:09.786829    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3ac547_fd72_474e_91ff_cc7123edffd7.slice/crio-30ebf88a2be53a4d52f34451dab3621912b867734b6ef7598f4dd11f373bf9f3 WatchSource:0}: Error finding container 30ebf88a2be53a4d52f34451dab3621912b867734b6ef7598f4dd11f373bf9f3: Status 404 returned error can't find the container with id 30ebf88a2be53a4d52f34451dab3621912b867734b6ef7598f4dd11f373bf9f3
Sep 12 01:45:10.094438 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.094288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:10.094438 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.094423    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:10.094586 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.094465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:10.094586 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.094479    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:12.094465224 +0000 UTC m=+6.021411092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:10.094586 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.094526    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:10.094586 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.094566    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:12.094555296 +0000 UTC m=+6.021501184 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:10.296565 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.296487    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:10.296781 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.296678    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:10.296781 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.296698    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:10.296781 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.296711    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:10.296781 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.296768    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:12.296749155 +0000 UTC m=+6.223695022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:10.508328 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.508259    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-11 01:40:08 +0000 UTC" deadline="2028-03-05 07:27:25.356123705 +0000 UTC"
Sep 12 01:45:10.508328 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.508299    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12965h42m14.847829306s"
Sep 12 01:45:10.549972 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.549906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:10.550138 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.550063    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:10.550510 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.550491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:10.550639 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:10.550618    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:10.574940 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.574901    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2" event={"ID":"498ffecf-2695-4c69-bf14-c6bf64599f72","Type":"ContainerStarted","Data":"c5659ec76871d102b397c9e22c92bbf38df91ecccd00957c97f4972d033e78af"}
Sep 12 01:45:10.580192 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.580163    2582 generic.go:356] "Generic (PLEG): container finished" podID="d6911f5f0618fc5e9567bb786e74c577" containerID="dbcb317cd237e9b08eeb52206b20374226be383f4d903de912a79347afffddeb" exitCode=0
Sep 12 01:45:10.580304 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.580242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"d6911f5f0618fc5e9567bb786e74c577","Type":"ContainerDied","Data":"dbcb317cd237e9b08eeb52206b20374226be383f4d903de912a79347afffddeb"}
Sep 12 01:45:10.602752 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.602710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5zgg" event={"ID":"1c3ac547-fd72-474e-91ff-cc7123edffd7","Type":"ContainerStarted","Data":"30ebf88a2be53a4d52f34451dab3621912b867734b6ef7598f4dd11f373bf9f3"}
Sep 12 01:45:10.632659 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.632629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerStarted","Data":"4a4a713b1766482f295903c3152f793bc41ea551da15a2ae58fe5f43a7930e96"}
Sep 12 01:45:10.645869 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.645708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"e15f6af2a0de5765a60a329c6baacd87f796b0a1d1be73eadedbc76a4e2523a7"}
Sep 12 01:45:10.659261 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.659234    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8dsz" event={"ID":"8d726d2d-af38-4e62-84f9-67bf9533132b","Type":"ContainerStarted","Data":"24272a07bc248bfdc2d3b52f0763785f7b1a12570dd9c964653945ace6b86fe2"}
Sep 12 01:45:10.682004 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.681977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4ndgk" event={"ID":"0c82b21e-bb74-4edf-9d48-e8bafbdc3037","Type":"ContainerStarted","Data":"054d87dc6807e729072cf0851fb2b52e5e657dd71c46e948729a4df77aca507c"}
Sep 12 01:45:10.688988 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.688936    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mvbd7" event={"ID":"2120641c-d0f4-41db-b3bc-734724f523b7","Type":"ContainerStarted","Data":"952922d3f8f51b56f127f1e4d5444323659039c85fc3dc4b6fe1c2400fbd81dd"}
Sep 12 01:45:10.707065 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.706778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" event={"ID":"8c31d8c8119d4fdd6c9aac0791f891f3","Type":"ContainerStarted","Data":"2b47b14946a9699ca155b637c944b8649c26e7d1992134908a07aa7a8ecede97"}
Sep 12 01:45:10.714675 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.714648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwflt" event={"ID":"440dbf68-d46f-4711-9c48-cbd0ede84229","Type":"ContainerStarted","Data":"99b1e06fa2866dff41f8160042994af77070625329ef3bd87e20d3a601258771"}
Sep 12 01:45:10.725343 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:10.725317    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbszw" event={"ID":"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99","Type":"ContainerStarted","Data":"1d4fac731768fc926434b9aa0fbaab71e9f626a7a928b0b7ba9a7df8d748dd95"}
Sep 12 01:45:11.549204 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:11.549171    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:11.549897 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:11.549297    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:11.749117 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:11.748113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" event={"ID":"d6911f5f0618fc5e9567bb786e74c577","Type":"ContainerStarted","Data":"3360e88a8a15179dca9d42cc4948a7d572f19c5c28fdf5574e2a1f0abb0d318b"}
Sep 12 01:45:11.763636 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:11.762536    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-81.ec2.internal" podStartSLOduration=4.76251682 podStartE2EDuration="4.76251682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:10.722154096 +0000 UTC m=+4.649099984" watchObservedRunningTime="2026-09-12 01:45:11.76251682 +0000 UTC m=+5.689462707"
Sep 12 01:45:12.119746 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:12.119661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:12.119746 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:12.119738    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:12.119959 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.119883    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:12.119959 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.119945    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:16.119927331 +0000 UTC m=+10.046873217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:12.120405 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.120367    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:12.120503 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.120421    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:16.120406592 +0000 UTC m=+10.047352472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:12.322368 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:12.321765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:12.322368 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.321928    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:12.322368 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.321952    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:12.322368 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.321964    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:12.322368 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.322021    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:16.322002513 +0000 UTC m=+10.248948411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:12.552067 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:12.551984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:12.552491 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.552123    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:12.552563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:12.552501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:12.552627 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:12.552606    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:13.549714 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:13.549683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:13.549908 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:13.549824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:14.550149 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:14.550121    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:14.550596 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:14.550241    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:14.552596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:14.552425    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:14.552596 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:14.552546    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:15.549089 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:15.549008    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:15.549230 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:15.549144    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:16.157964 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:16.157915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:16.158438 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:16.158006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:16.158438 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.158099    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:16.158438 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.158129    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:16.158438 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.158166    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:24.158150166 +0000 UTC m=+18.085096029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:16.158438 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.158184    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:24.158175602 +0000 UTC m=+18.085121466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:16.359902 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:16.359258    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:16.359902 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.359477    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:16.359902 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.359499    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:16.359902 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.359512    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:16.359902 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.359569    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:24.359549313 +0000 UTC m=+18.286495200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:16.550457 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:16.550388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:16.550615 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.550490    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:16.550935 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:16.550910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:16.551084 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:16.551009    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:17.549057 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:17.549012    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:17.549652 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:17.549144    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:18.551445 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:18.551410    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:18.551796 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:18.551553    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:18.551964 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:18.551938    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:18.552098 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:18.552066    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:19.549248 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:19.549215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:19.549389 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:19.549315    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:20.548913 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:20.548829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:20.549410 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:20.548951    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:20.549410 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:20.549004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:20.549410 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:20.549113    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:21.549387 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:21.549349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:21.549802 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:21.549458    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:22.552123 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:22.552097    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:22.552565 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:22.552213    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:22.552565 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:22.552300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:22.552565 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:22.552419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:23.549534 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:23.549504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:23.549713 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:23.549615    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:24.212616 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:24.212577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:24.213191 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:24.212655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:24.213191 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.212748    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:24.213191 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.212830    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:45:40.21280655 +0000 UTC m=+34.139752429 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:24.213191 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.212759    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:24.213191 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.212897    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:40.212883343 +0000 UTC m=+34.139829224 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:24.414020 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:24.413985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:24.414209 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.414156    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:24.414209 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.414176    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:24.414209 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.414186    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:24.414373 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.414239    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:40.414224434 +0000 UTC m=+34.341170311 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:24.549872 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:24.549786    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:24.550010 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:24.549795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:24.550010 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.549929    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:24.550133 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:24.550024    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:25.549769 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:25.549730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:25.550229 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:25.549865    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:26.550276 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:26.550236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:26.550722 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:26.550351    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:26.550722 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:26.550439    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:26.550722 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:26.550591    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:27.549144 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.548974    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:27.549233 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:27.549217    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:27.782748 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.782723    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x5zgg" event={"ID":"1c3ac547-fd72-474e-91ff-cc7123edffd7","Type":"ContainerStarted","Data":"9fc265e6220df18b95b24c915bdbcbdca0bc9c6577d40697f805faea302bb4cb"}
Sep 12 01:45:27.784102 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.784080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerStarted","Data":"a43f11252dad069c4715d66b7f184b224a5ff35c9e50a9361bec6abab7d26e05"}
Sep 12 01:45:27.785879 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.785856    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"4c6eca2ff4f56572dcf9293bbd1da1ccbe1d994fac3ed16be5b25d9a5c9bcc3a"}
Sep 12 01:45:27.785978 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.785928    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"4ca85421927924850eceeb3f86b947071006ba540f1bffb65a4fb348f39b2a47"}
Sep 12 01:45:27.787516 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.787495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-b8dsz" event={"ID":"8d726d2d-af38-4e62-84f9-67bf9533132b","Type":"ContainerStarted","Data":"d21db1e3ca797505d84c05ea66f66a1c6aa64e2d04d1f7c8680a776118caf83a"}
Sep 12 01:45:27.788797 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.788780    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4ndgk" event={"ID":"0c82b21e-bb74-4edf-9d48-e8bafbdc3037","Type":"ContainerStarted","Data":"35b12b968f39fef84e8466715ba14a5f146770cd287933f29c6a968521f08565"}
Sep 12 01:45:27.790083 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.790063    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mvbd7" event={"ID":"2120641c-d0f4-41db-b3bc-734724f523b7","Type":"ContainerStarted","Data":"085ae57beb884621e5d095632f016271a317a9850d9e351fa1e941cffdd43f19"}
Sep 12 01:45:27.791378 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.791361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2" event={"ID":"498ffecf-2695-4c69-bf14-c6bf64599f72","Type":"ContainerStarted","Data":"291a5f68bc4e501dc4e540b9166a0eedb8540440a4339d490b85e9bb923ec291"}
Sep 12 01:45:27.796163 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.796127    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-81.ec2.internal" podStartSLOduration=20.796115918 podStartE2EDuration="20.796115918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:11.763588762 +0000 UTC m=+5.690534648" watchObservedRunningTime="2026-09-12 01:45:27.796115918 +0000 UTC m=+21.723061801"
Sep 12 01:45:27.796373 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.796355    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x5zgg" podStartSLOduration=7.884172185 podStartE2EDuration="21.796350317s" totalImagesPullingTime="13.912178132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:27.795824905 +0000 UTC m=+21.722770787" watchObservedRunningTime="2026-09-12 01:45:27.796350317 +0000 UTC m=+21.723296234"
Sep 12 01:45:27.808262 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.808224    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mvbd7" podStartSLOduration=4.242300857 podStartE2EDuration="21.808212674s" totalImagesPullingTime="17.565911817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:27.808115445 +0000 UTC m=+21.735061330" watchObservedRunningTime="2026-09-12 01:45:27.808212674 +0000 UTC m=+21.735158560"
Sep 12 01:45:27.822706 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.822660    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-b8dsz" podStartSLOduration=4.257179534 podStartE2EDuration="21.822644949s" totalImagesPullingTime="17.565465415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:27.822581321 +0000 UTC m=+21.749527206" watchObservedRunningTime="2026-09-12 01:45:27.822644949 +0000 UTC m=+21.749590836"
Sep 12 01:45:27.852272 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:27.852230    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4ndgk" podStartSLOduration=4.287317812 podStartE2EDuration="21.85221623s" totalImagesPullingTime="17.564898418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:27.851742663 +0000 UTC m=+21.778688550" watchObservedRunningTime="2026-09-12 01:45:27.85221623 +0000 UTC m=+21.779162113"
Sep 12 01:45:28.553293 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.553132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:28.553424 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.553132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:28.553424 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:28.553400    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:28.553575 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:28.553525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:28.613802 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.613766    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 12 01:45:28.664474 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.664381    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-12T01:45:28.613783026Z","UUID":"80deee3f-a98b-4db9-85f3-6083299f8e98","Handler":null,"Name":"","Endpoint":""}
Sep 12 01:45:28.665521 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.665506    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 12 01:45:28.665587 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.665531    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 12 01:45:28.794908 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.794844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwflt" event={"ID":"440dbf68-d46f-4711-9c48-cbd0ede84229","Type":"ContainerStarted","Data":"c7346b7958713f4980471623d31c9903f2acf261275edd2dabddf366845723cf"}
Sep 12 01:45:28.796386 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.796363    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wbszw" event={"ID":"5cb6bff6-c91f-4c5a-aeb3-86e7e61a0c99","Type":"ContainerStarted","Data":"d28d68716b61f30ebdba9b7e549b9b9e4418501395bf8ec7caeed7f8caf9dba9"}
Sep 12 01:45:28.798206 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.798185    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2" event={"ID":"498ffecf-2695-4c69-bf14-c6bf64599f72","Type":"ContainerStarted","Data":"df293701d4ad27a50c1c15949f9f696286292d3968bc4893cd87cd1ff6904592"}
Sep 12 01:45:28.799563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.799544    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="a43f11252dad069c4715d66b7f184b224a5ff35c9e50a9361bec6abab7d26e05" exitCode=0
Sep 12 01:45:28.799640 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.799619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"a43f11252dad069c4715d66b7f184b224a5ff35c9e50a9361bec6abab7d26e05"}
Sep 12 01:45:28.802484 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.802468    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:45:28.802806 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.802786    2582 generic.go:356] "Generic (PLEG): container finished" podID="685ab9df-4254-4f9e-8208-30d659158759" containerID="4c6eca2ff4f56572dcf9293bbd1da1ccbe1d994fac3ed16be5b25d9a5c9bcc3a" exitCode=1
Sep 12 01:45:28.802902 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.802883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerDied","Data":"4c6eca2ff4f56572dcf9293bbd1da1ccbe1d994fac3ed16be5b25d9a5c9bcc3a"}
Sep 12 01:45:28.802954 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.802943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"2c726e270ccf91716181422f3a6a8fc6425c237c86d95affa9a48cf1fa1788ea"}
Sep 12 01:45:28.802988 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.802975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"f4cb78325f0e88083e4d058fc4f4cc0e286c363039cbd092e0262ade7c766f38"}
Sep 12 01:45:28.803020 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.803005    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"a2684f9a6f7007a78ad59d24bb3c14cc83f09107cdc0e37a4db46d805cacc310"}
Sep 12 01:45:28.803073 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.803031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"d30ce0d66a2d0a33f951248563741eb4dd0083971f1d02820d6a57dee617ee16"}
Sep 12 01:45:28.811808 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.811774    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nwflt" podStartSLOduration=4.662112994 podStartE2EDuration="22.811762346s" totalImagesPullingTime="18.149649352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:28.811183678 +0000 UTC m=+22.738129567" watchObservedRunningTime="2026-09-12 01:45:28.811762346 +0000 UTC m=+22.738708232"
Sep 12 01:45:28.840806 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:28.840773    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wbszw" podStartSLOduration=5.27171318 podStartE2EDuration="22.840761702s" totalImagesPullingTime="17.569048522s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:28.840677967 +0000 UTC m=+22.767623853" watchObservedRunningTime="2026-09-12 01:45:28.840761702 +0000 UTC m=+22.767707588"
Sep 12 01:45:29.549259 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.549223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:29.549430 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:29.549321    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:29.595646 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.595616    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:29.596543 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.596511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:29.807365 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.807281    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2" event={"ID":"498ffecf-2695-4c69-bf14-c6bf64599f72","Type":"ContainerStarted","Data":"8532a1801c068f25fd955f960d1557fa10e56456aaae77c806c9312eb2dee5b2"}
Sep 12 01:45:29.807930 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.807909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:29.808338 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.808323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x5zgg"
Sep 12 01:45:29.823255 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:29.823210    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c5jb2" podStartSLOduration=4.383530454 podStartE2EDuration="23.823196517s" totalImagesPullingTime="19.439666063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:29.822954161 +0000 UTC m=+23.749900047" watchObservedRunningTime="2026-09-12 01:45:29.823196517 +0000 UTC m=+23.750142420"
Sep 12 01:45:30.549290 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:30.549088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:30.549290 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:30.549088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:30.549447 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:30.549378    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:30.549447 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:30.549412    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:30.812619 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:30.812540    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:45:30.812982 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:30.812942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"c0605419313432e713a50e2473faeaec140a934669a965e51c121c1c3696f899"}
Sep 12 01:45:31.549069 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:31.549023    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:31.549264 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:31.549150    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:32.549346 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:32.549281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:32.549690 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:32.549281    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:32.549690 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:32.549396    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:32.549690 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:32.549459    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:33.549792 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:33.549752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:33.550232 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:33.549884    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:34.549317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.549163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:34.549418 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.549163    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:34.549531 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:34.549509    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:34.549567 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:34.549388    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:34.823146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.823063    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="ae7eb15cc6bb8b874f9b3950da9a11438573e0cd42bc78b74413fa804560f71d" exitCode=0
Sep 12 01:45:34.823638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.823144    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"ae7eb15cc6bb8b874f9b3950da9a11438573e0cd42bc78b74413fa804560f71d"}
Sep 12 01:45:34.826915 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.826891    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:45:34.827251 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.827232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"a4fa8a07c4b59915b89096d0b379510268fbc650cc10f3892e3e2bc6e2a8cf50"}
Sep 12 01:45:34.827587 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.827560    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:34.827764 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.827607    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:34.827764 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.827724    2582 scope.go:117] "RemoveContainer" containerID="4c6eca2ff4f56572dcf9293bbd1da1ccbe1d994fac3ed16be5b25d9a5c9bcc3a"
Sep 12 01:45:34.842747 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.842723    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:34.842872 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:34.842859    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:35.549898 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.549860    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:35.550090 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:35.549987    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:35.788517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.788330    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzpgh"]
Sep 12 01:45:35.788641 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.788606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:35.788707 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:35.788689    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:35.792128 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.792105    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk2nm"]
Sep 12 01:45:35.792248 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.792227    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:35.792348 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:35.792330    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:35.792669 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.792649    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb84r"]
Sep 12 01:45:35.831822 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.831766    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="aacb23cb91dd45da13083a11ef33b1155e23ea44d3ed46e6249258d5c76a30ea" exitCode=0
Sep 12 01:45:35.832409 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.831838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"aacb23cb91dd45da13083a11ef33b1155e23ea44d3ed46e6249258d5c76a30ea"}
Sep 12 01:45:35.836138 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.836122    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:45:35.836553 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.836528    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" event={"ID":"685ab9df-4254-4f9e-8208-30d659158759","Type":"ContainerStarted","Data":"d6a34040243a6416497778bdd134877d44cb31f01b801b1a6be112dda29a1a72"}
Sep 12 01:45:35.836553 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.836548    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:35.836668 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:35.836636    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:35.836668 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.836640    2582 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 12 01:45:35.876052 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:35.876009    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx" podStartSLOduration=12.277810036 podStartE2EDuration="29.875996856s" totalImagesPullingTime="17.59818682s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:35.875800795 +0000 UTC m=+29.802746694" watchObservedRunningTime="2026-09-12 01:45:35.875996856 +0000 UTC m=+29.802942742"
Sep 12 01:45:36.475470 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:36.475439    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:45:36.841608 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:36.841574    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="19f26e5fab54f0fd29ce744eb4b5dde666cb2db6af0aa7de2859de0782f827da" exitCode=0
Sep 12 01:45:36.841960 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:36.841656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"19f26e5fab54f0fd29ce744eb4b5dde666cb2db6af0aa7de2859de0782f827da"}
Sep 12 01:45:37.549131 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:37.549085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:37.549131 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:37.549109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:37.549370 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:37.549111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:37.549370 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:37.549202    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:37.549370 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:37.549270    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:37.549618 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:37.549595    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:39.549750 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:39.549620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:39.549750 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:39.549647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:39.549750 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:39.549654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:39.550316 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:39.549758    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pb84r" podUID="db1bbe8c-18c3-48f1-992c-282a06e7f413"
Sep 12 01:45:39.550316 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:39.549879    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dzpgh" podUID="d7fff888-e442-4677-a3c9-9b5d490bb332"
Sep 12 01:45:39.550316 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:39.549958    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xk2nm" podUID="4db996df-7bb9-42a4-9c57-497e46831b9e"
Sep 12 01:45:40.161426 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.161399    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mvbd7_2120641c-d0f4-41db-b3bc-734724f523b7/dns-node-resolver/0.log"
Sep 12 01:45:40.238988 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.238953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:40.239196 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.239023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:40.239196 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.239132    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:40.239196 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.239161    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:40.239382 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.239205    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret podName:d7fff888-e442-4677-a3c9-9b5d490bb332 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:12.239184439 +0000 UTC m=+66.166130307 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret") pod "global-pull-secret-syncer-dzpgh" (UID: "d7fff888-e442-4677-a3c9-9b5d490bb332") : object "kube-system"/"original-pull-secret" not registered
Sep 12 01:45:40.239382 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.239226    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs podName:4db996df-7bb9-42a4-9c57-497e46831b9e nodeName:}" failed. No retries permitted until 2026-09-12 01:46:12.239216451 +0000 UTC m=+66.166162315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs") pod "network-metrics-daemon-xk2nm" (UID: "4db996df-7bb9-42a4-9c57-497e46831b9e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 12 01:45:40.440419 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.440350    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:40.440568 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.440511    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 12 01:45:40.440568 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.440531    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 12 01:45:40.440568 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.440541    2582 projected.go:196] Error preparing data for projected volume kube-api-access-btc88 for pod openshift-network-diagnostics/network-check-target-pb84r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:40.440688 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:40.440596    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88 podName:db1bbe8c-18c3-48f1-992c-282a06e7f413 nodeName:}" failed. No retries permitted until 2026-09-12 01:46:12.44057815 +0000 UTC m=+66.367524017 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-btc88" (UniqueName: "kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88") pod "network-check-target-pb84r" (UID: "db1bbe8c-18c3-48f1-992c-282a06e7f413") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 12 01:45:40.918730 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.918701    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-81.ec2.internal" event="NodeReady"
Sep 12 01:45:40.919288 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.918835    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 12 01:45:40.963106 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:40.963079    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ql2vm"]
Sep 12 01:45:41.009468 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.009443    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-km8s9"]
Sep 12 01:45:41.009657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.009533    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.012385 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.012365    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 12 01:45:41.012528 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.012406    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5l94s\""
Sep 12 01:45:41.013658 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.013319    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 12 01:45:41.013658 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.013394    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 12 01:45:41.026977 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.026959    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ql2vm"]
Sep 12 01:45:41.027073 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.026987    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n8kws"]
Sep 12 01:45:41.027121 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.027087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.029225 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.029206    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 12 01:45:41.029301 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.029206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 12 01:45:41.029443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.029431    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 12 01:45:41.029504 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.029489    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 12 01:45:41.029974 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.029961    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9fddj\""
Sep 12 01:45:41.051210 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.051185    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-km8s9"]
Sep 12 01:45:41.051295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.051227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8kws"]
Sep 12 01:45:41.051295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.051284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.054694 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.054676    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gmqgn\""
Sep 12 01:45:41.054785 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.054722    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 12 01:45:41.054785 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.054681    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 12 01:45:41.144757 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwfk\" (UniqueName: \"kubernetes.io/projected/ede064d1-2366-41ee-a796-62ecab84e0f8-kube-api-access-scwfk\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.144757 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7b86\" (UniqueName: \"kubernetes.io/projected/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-api-access-h7b86\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.144945 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144803    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.144945 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e45e25-6a1a-4019-a55c-b4a68713ced9-data-volume\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.144945 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1e45e25-6a1a-4019-a55c-b4a68713ced9-crio-socket\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.144945 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144931    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.145152 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144965    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-cert\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.145152 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.144984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbrbj\" (UniqueName: \"kubernetes.io/projected/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-kube-api-access-zbrbj\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.145152 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.145010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ede064d1-2366-41ee-a796-62ecab84e0f8-config-volume\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.145152 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.145056    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.145152 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.145070    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede064d1-2366-41ee-a796-62ecab84e0f8-tmp-dir\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.245657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.245657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e45e25-6a1a-4019-a55c-b4a68713ced9-data-volume\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.245657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1e45e25-6a1a-4019-a55c-b4a68713ced9-crio-socket\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.245702    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.245760    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.245785    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:41.745763984 +0000 UTC m=+35.672709874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : configmap "kube-rbac-proxy" not found
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.245830    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:41.745811026 +0000 UTC m=+35.672756890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-cert\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.245894 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245889    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbrbj\" (UniqueName: \"kubernetes.io/projected/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-kube-api-access-zbrbj\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ede064d1-2366-41ee-a796-62ecab84e0f8-config-volume\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1e45e25-6a1a-4019-a55c-b4a68713ced9-crio-socket\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1e45e25-6a1a-4019-a55c-b4a68713ced9-data-volume\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.245976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede064d1-2366-41ee-a796-62ecab84e0f8-tmp-dir\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.246004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwfk\" (UniqueName: \"kubernetes.io/projected/ede064d1-2366-41ee-a796-62ecab84e0f8-kube-api-access-scwfk\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.246030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7b86\" (UniqueName: \"kubernetes.io/projected/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-api-access-h7b86\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.246074    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 01:45:41.246260 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.246130    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls podName:ede064d1-2366-41ee-a796-62ecab84e0f8 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:41.746111958 +0000 UTC m=+35.673057836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls") pod "dns-default-n8kws" (UID: "ede064d1-2366-41ee-a796-62ecab84e0f8") : secret "dns-default-metrics-tls" not found
Sep 12 01:45:41.246561 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.246359    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede064d1-2366-41ee-a796-62ecab84e0f8-tmp-dir\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.246561 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.246442    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ede064d1-2366-41ee-a796-62ecab84e0f8-config-volume\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.250110 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.250091    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-cert\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.254018 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.253994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbrbj\" (UniqueName: \"kubernetes.io/projected/1d8a84f0-e955-4ca5-a08d-b2391e5ca88c-kube-api-access-zbrbj\") pod \"ingress-canary-ql2vm\" (UID: \"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c\") " pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.258177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.258151    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7b86\" (UniqueName: \"kubernetes.io/projected/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-api-access-h7b86\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.258295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.258222    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwfk\" (UniqueName: \"kubernetes.io/projected/ede064d1-2366-41ee-a796-62ecab84e0f8-kube-api-access-scwfk\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.320168 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.320132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ql2vm"
Sep 12 01:45:41.344782 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.344762    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4ndgk_0c82b21e-bb74-4edf-9d48-e8bafbdc3037/node-ca/0.log"
Sep 12 01:45:41.388522 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.388490    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-gphdf"]
Sep 12 01:45:41.407715 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.407684    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-gphdf"]
Sep 12 01:45:41.407846 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.407731    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.410146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.410117    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 12 01:45:41.410274 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.410189    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 12 01:45:41.410274 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.410264    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 12 01:45:41.410384 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.410265    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lkqwx\""
Sep 12 01:45:41.410440 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.410388    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.549052    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-cabundle\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.549342    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-key\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.549367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb72g\" (UniqueName: \"kubernetes.io/projected/fcdb19fb-18ea-4e67-8808-47a25febb304-kube-api-access-vb72g\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.549583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.549975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:45:41.550855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.550293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553176    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553358    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553370    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553477    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fqtlc\""
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553596    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmzff\""
Sep 12 01:45:41.553906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.553620    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 01:45:41.649754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.649729    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-cabundle\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.649859 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.649789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-key\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.649859 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.649807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb72g\" (UniqueName: \"kubernetes.io/projected/fcdb19fb-18ea-4e67-8808-47a25febb304-kube-api-access-vb72g\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.650353 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.650332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-cabundle\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.652563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.652541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/fcdb19fb-18ea-4e67-8808-47a25febb304-signing-key\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.656751 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.656732    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb72g\" (UniqueName: \"kubernetes.io/projected/fcdb19fb-18ea-4e67-8808-47a25febb304-kube-api-access-vb72g\") pod \"service-ca-568869fdc5-gphdf\" (UID: \"fcdb19fb-18ea-4e67-8808-47a25febb304\") " pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.716559 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.716523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-gphdf"
Sep 12 01:45:41.751094 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.751071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:41.751219 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751201    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 01:45:41.751286 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.751227    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.751286 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751283    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls podName:ede064d1-2366-41ee-a796-62ecab84e0f8 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.751268006 +0000 UTC m=+36.678213874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls") pod "dns-default-n8kws" (UID: "ede064d1-2366-41ee-a796-62ecab84e0f8") : secret "dns-default-metrics-tls" not found
Sep 12 01:45:41.751391 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751323    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 01:45:41.751436 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751393    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.751374402 +0000 UTC m=+36.678320282 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : configmap "kube-rbac-proxy" not found
Sep 12 01:45:41.751485 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.751429    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:41.751583 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751564    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:41.751637 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:41.751626    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:42.751610684 +0000 UTC m=+36.678556568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:41.763950 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.763928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ql2vm"]
Sep 12 01:45:41.805876 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:41.805180    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8a84f0_e955_4ca5_a08d_b2391e5ca88c.slice/crio-82c31eae6a98769ca09929bd98d7b920146ca0c1116299e0d87f35d640a70aeb WatchSource:0}: Error finding container 82c31eae6a98769ca09929bd98d7b920146ca0c1116299e0d87f35d640a70aeb: Status 404 returned error can't find the container with id 82c31eae6a98769ca09929bd98d7b920146ca0c1116299e0d87f35d640a70aeb
Sep 12 01:45:41.854611 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.854580    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ql2vm" event={"ID":"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c","Type":"ContainerStarted","Data":"82c31eae6a98769ca09929bd98d7b920146ca0c1116299e0d87f35d640a70aeb"}
Sep 12 01:45:41.857242 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.857221    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="9dcaf4f75402751bec99117fb8e2cc625052f30b85f92d9756c908f0930674ca" exitCode=0
Sep 12 01:45:41.857335 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:41.857272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"9dcaf4f75402751bec99117fb8e2cc625052f30b85f92d9756c908f0930674ca"}
Sep 12 01:45:42.023955 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.023928    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-gphdf"]
Sep 12 01:45:42.026984 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:42.026961    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdb19fb_18ea_4e67_8808_47a25febb304.slice/crio-944f5c160b3c099bf2d65a33a77e06d4a2e52d79180b652e801e3fd06016b6ff WatchSource:0}: Error finding container 944f5c160b3c099bf2d65a33a77e06d4a2e52d79180b652e801e3fd06016b6ff: Status 404 returned error can't find the container with id 944f5c160b3c099bf2d65a33a77e06d4a2e52d79180b652e801e3fd06016b6ff
Sep 12 01:45:42.761709 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.761470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:42.761709 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.761565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:42.761709 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.761615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:42.761709 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761660    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 01:45:42.762072 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761742    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls podName:ede064d1-2366-41ee-a796-62ecab84e0f8 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:44.761721635 +0000 UTC m=+38.688667514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls") pod "dns-default-n8kws" (UID: "ede064d1-2366-41ee-a796-62ecab84e0f8") : secret "dns-default-metrics-tls" not found
Sep 12 01:45:42.762072 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761765    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:42.762072 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761798    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 12 01:45:42.762072 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761815    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:44.761800301 +0000 UTC m=+38.688746190 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:42.762072 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:42.761842    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:44.761831029 +0000 UTC m=+38.688776913 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : configmap "kube-rbac-proxy" not found
Sep 12 01:45:42.864688 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.864651    2582 generic.go:356] "Generic (PLEG): container finished" podID="e2d2ce75-d3cd-41cc-8c67-70c3deeb0196" containerID="6c4d8f65e8226e71b03e7a4c50e36177c087b5cf7dc1a634edfa65e8e2edcb01" exitCode=0
Sep 12 01:45:42.864859 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.864759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerDied","Data":"6c4d8f65e8226e71b03e7a4c50e36177c087b5cf7dc1a634edfa65e8e2edcb01"}
Sep 12 01:45:42.866901 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:42.866859    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-gphdf" event={"ID":"fcdb19fb-18ea-4e67-8808-47a25febb304","Type":"ContainerStarted","Data":"944f5c160b3c099bf2d65a33a77e06d4a2e52d79180b652e801e3fd06016b6ff"}
Sep 12 01:45:43.874192 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:43.873947    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2blc" event={"ID":"e2d2ce75-d3cd-41cc-8c67-70c3deeb0196","Type":"ContainerStarted","Data":"496615f7765be8058a9fa6b6f755df11ec649bd369a375c03f2142a7a3778955"}
Sep 12 01:45:43.897225 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:43.897167    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r2blc" podStartSLOduration=8.793087147 podStartE2EDuration="37.897147267s" totalImagesPullingTime="29.10406012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:43.895624267 +0000 UTC m=+37.822570154" watchObservedRunningTime="2026-09-12 01:45:43.897147267 +0000 UTC m=+37.824093153"
Sep 12 01:45:44.777487 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:44.777453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:44.777640 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:44.777515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:44.777640 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:44.777557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:44.777640 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:44.777614    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:44.777842 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:44.777688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:48.777671852 +0000 UTC m=+42.704617717 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:44.777842 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:44.777614    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 01:45:44.777842 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:44.777745    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls podName:ede064d1-2366-41ee-a796-62ecab84e0f8 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:48.777733852 +0000 UTC m=+42.704679731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls") pod "dns-default-n8kws" (UID: "ede064d1-2366-41ee-a796-62ecab84e0f8") : secret "dns-default-metrics-tls" not found
Sep 12 01:45:44.778012 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:44.777996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1e45e25-6a1a-4019-a55c-b4a68713ced9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:45.880155 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:45.880113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-gphdf" event={"ID":"fcdb19fb-18ea-4e67-8808-47a25febb304","Type":"ContainerStarted","Data":"6c5800d7a08c5e791673f5f007c8f24fee5c2199b76d5cf7eabbe78f92bfdf35"}
Sep 12 01:45:45.881709 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:45.881685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ql2vm" event={"ID":"1d8a84f0-e955-4ca5-a08d-b2391e5ca88c","Type":"ContainerStarted","Data":"8dff84984f13f4b9c692389b970caa854c39a50dd3d9c5e56ac1ea4b69ee58cc"}
Sep 12 01:45:45.896221 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:45.896171    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-gphdf" podStartSLOduration=2.028181809 podStartE2EDuration="4.896154553s" totalImagesPullingTime="2.867972744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:45.895613281 +0000 UTC m=+39.822559167" watchObservedRunningTime="2026-09-12 01:45:45.896154553 +0000 UTC m=+39.823100440"
Sep 12 01:45:45.911397 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:45.911357    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ql2vm" podStartSLOduration=2.815480015 podStartE2EDuration="5.911345053s" totalImagesPullingTime="3.095865038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:45.910276271 +0000 UTC m=+39.837222169" watchObservedRunningTime="2026-09-12 01:45:45.911345053 +0000 UTC m=+39.838290933"
Sep 12 01:45:48.813246 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:48.813212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:48.813644 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:48.813283    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:48.813644 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:48.813367    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 12 01:45:48.813644 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:48.813426    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls podName:ede064d1-2366-41ee-a796-62ecab84e0f8 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:56.813412575 +0000 UTC m=+50.740358452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls") pod "dns-default-n8kws" (UID: "ede064d1-2366-41ee-a796-62ecab84e0f8") : secret "dns-default-metrics-tls" not found
Sep 12 01:45:48.813644 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:48.813372    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:48.813644 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:45:48.813488    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls podName:a1e45e25-6a1a-4019-a55c-b4a68713ced9 nodeName:}" failed. No retries permitted until 2026-09-12 01:45:56.813477326 +0000 UTC m=+50.740423192 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-km8s9" (UID: "a1e45e25-6a1a-4019-a55c-b4a68713ced9") : secret "insights-runtime-extractor-tls" not found
Sep 12 01:45:56.870717 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.870680    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:56.871143 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.870741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:56.873942 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.873914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1e45e25-6a1a-4019-a55c-b4a68713ced9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-km8s9\" (UID: \"a1e45e25-6a1a-4019-a55c-b4a68713ced9\") " pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:56.874074 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.873915    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ede064d1-2366-41ee-a796-62ecab84e0f8-metrics-tls\") pod \"dns-default-n8kws\" (UID: \"ede064d1-2366-41ee-a796-62ecab84e0f8\") " pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:56.934784 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.934760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-km8s9"
Sep 12 01:45:56.962061 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:56.962027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:57.231455 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:57.231378    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-km8s9"]
Sep 12 01:45:57.234404 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:57.234375    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e45e25_6a1a_4019_a55c_b4a68713ced9.slice/crio-9002a3215f8dfae8ee941573c2999be23a1b9dead5d4cc388cfb94ab663490f5 WatchSource:0}: Error finding container 9002a3215f8dfae8ee941573c2999be23a1b9dead5d4cc388cfb94ab663490f5: Status 404 returned error can't find the container with id 9002a3215f8dfae8ee941573c2999be23a1b9dead5d4cc388cfb94ab663490f5
Sep 12 01:45:57.256921 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:57.256898    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n8kws"]
Sep 12 01:45:57.259649 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:45:57.259622    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede064d1_2366_41ee_a796_62ecab84e0f8.slice/crio-564d338c893cc41fb9aa56c023772aee0682ed5c0f099b51d74f300d75ac65da WatchSource:0}: Error finding container 564d338c893cc41fb9aa56c023772aee0682ed5c0f099b51d74f300d75ac65da: Status 404 returned error can't find the container with id 564d338c893cc41fb9aa56c023772aee0682ed5c0f099b51d74f300d75ac65da
Sep 12 01:45:57.914866 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:57.914818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-km8s9" event={"ID":"a1e45e25-6a1a-4019-a55c-b4a68713ced9","Type":"ContainerStarted","Data":"3bbea6237fb897bf5444469e8a7d56208dac415415d74d7dd62492012f754fb1"}
Sep 12 01:45:57.915337 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:57.914905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-km8s9" event={"ID":"a1e45e25-6a1a-4019-a55c-b4a68713ced9","Type":"ContainerStarted","Data":"9002a3215f8dfae8ee941573c2999be23a1b9dead5d4cc388cfb94ab663490f5"}
Sep 12 01:45:57.916489 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:57.916454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8kws" event={"ID":"ede064d1-2366-41ee-a796-62ecab84e0f8","Type":"ContainerStarted","Data":"564d338c893cc41fb9aa56c023772aee0682ed5c0f099b51d74f300d75ac65da"}
Sep 12 01:45:58.921506 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:58.921468    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-km8s9" event={"ID":"a1e45e25-6a1a-4019-a55c-b4a68713ced9","Type":"ContainerStarted","Data":"0bcff37e41e2b32df77c94f4d5818b982786f049fdfc69d9ef893d476793c9b9"}
Sep 12 01:45:59.926488 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.926461    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8kws" event={"ID":"ede064d1-2366-41ee-a796-62ecab84e0f8","Type":"ContainerStarted","Data":"cb6bb872220c3725b534a25205a719bf340e0df8ede2e79f24def0b6d4ef3712"}
Sep 12 01:45:59.926804 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.926509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n8kws" event={"ID":"ede064d1-2366-41ee-a796-62ecab84e0f8","Type":"ContainerStarted","Data":"5dada0199801183780ec9965ff317d46f266d78671e7f9457d4c89e7b558e49d"}
Sep 12 01:45:59.926804 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.926568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n8kws"
Sep 12 01:45:59.928533 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.928510    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-km8s9" event={"ID":"a1e45e25-6a1a-4019-a55c-b4a68713ced9","Type":"ContainerStarted","Data":"57e2c24090016e4d082bf9bf2e2358708b06e562d753f78b68e8cc4fd0856b90"}
Sep 12 01:45:59.944189 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.944149    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n8kws" podStartSLOduration=18.238301814 podStartE2EDuration="19.944136928s" totalImagesPullingTime="1.705835114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:59.94267104 +0000 UTC m=+53.869616926" watchObservedRunningTime="2026-09-12 01:45:59.944136928 +0000 UTC m=+53.871082814"
Sep 12 01:45:59.957896 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:45:59.957856    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-km8s9" podStartSLOduration=17.597759535 podStartE2EDuration="19.957844337s" totalImagesPullingTime="2.360084802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:45:59.957355142 +0000 UTC m=+53.884301028" watchObservedRunningTime="2026-09-12 01:45:59.957844337 +0000 UTC m=+53.884790222"
Sep 12 01:46:02.134385 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.134355    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"]
Sep 12 01:46:02.136401 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.136382    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.136999 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.136981    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"]
Sep 12 01:46:02.138534 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.138518    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"]
Sep 12 01:46:02.138673 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.138651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.138825 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.138799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 12 01:46:02.138922 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.138799    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pccln\""
Sep 12 01:46:02.139269 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.139254    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 12 01:46:02.140083 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.140066    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 12 01:46:02.140169 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.140117    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 12 01:46:02.140223 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.140209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.141166 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.141152    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 12 01:46:02.142507 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.142492    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 12 01:46:02.142698 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.142685    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 12 01:46:02.142995 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.142978    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 12 01:46:02.143091 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.143064    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 12 01:46:02.152776 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.152756    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"]
Sep 12 01:46:02.153516 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.153495    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"]
Sep 12 01:46:02.154112 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.154095    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"]
Sep 12 01:46:02.204925 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.204898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.205022 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.204934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4631961e-5c67-4a93-a16f-fc750132a627-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.205022 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.204954    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e79d99-63a6-4620-9b30-455f7e1b63fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.205022 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.204994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9rrs\" (UniqueName: \"kubernetes.io/projected/04e79d99-63a6-4620-9b30-455f7e1b63fe-kube-api-access-d9rrs\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.205146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4631961e-5c67-4a93-a16f-fc750132a627-tmp\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.205146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2caef38d-9c72-4104-9048-971011881e28-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.205146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205088    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.205234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqxv\" (UniqueName: \"kubernetes.io/projected/2caef38d-9c72-4104-9048-971011881e28-kube-api-access-6xqxv\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.205234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205188    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.205234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p48rb\" (UniqueName: \"kubernetes.io/projected/4631961e-5c67-4a93-a16f-fc750132a627-kube-api-access-p48rb\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.205234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.205224    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.305942 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.305914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e79d99-63a6-4620-9b30-455f7e1b63fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.305942 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.305943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9rrs\" (UniqueName: \"kubernetes.io/projected/04e79d99-63a6-4620-9b30-455f7e1b63fe-kube-api-access-d9rrs\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.306094 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.305960    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4631961e-5c67-4a93-a16f-fc750132a627-tmp\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.306094 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.305976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2caef38d-9c72-4104-9048-971011881e28-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.306211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.306295 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqxv\" (UniqueName: \"kubernetes.io/projected/2caef38d-9c72-4104-9048-971011881e28-kube-api-access-6xqxv\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.306352 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4631961e-5c67-4a93-a16f-fc750132a627-tmp\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.306352 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.306452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p48rb\" (UniqueName: \"kubernetes.io/projected/4631961e-5c67-4a93-a16f-fc750132a627-kube-api-access-p48rb\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.306452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.306452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.306602 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4631961e-5c67-4a93-a16f-fc750132a627-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.306706 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.306676    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04e79d99-63a6-4620-9b30-455f7e1b63fe-additional-service-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.308909 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.308880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.309115 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.309096    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-ca\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.309208 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.309188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.309266 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.309250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04e79d99-63a6-4620-9b30-455f7e1b63fe-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.309304 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.309263    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2caef38d-9c72-4104-9048-971011881e28-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.309638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.309619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4631961e-5c67-4a93-a16f-fc750132a627-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.315532 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.315505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqxv\" (UniqueName: \"kubernetes.io/projected/2caef38d-9c72-4104-9048-971011881e28-kube-api-access-6xqxv\") pod \"managed-serviceaccount-addon-agent-67b578c6fb-gp92h\" (UID: \"2caef38d-9c72-4104-9048-971011881e28\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.318952 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.318927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p48rb\" (UniqueName: \"kubernetes.io/projected/4631961e-5c67-4a93-a16f-fc750132a627-kube-api-access-p48rb\") pod \"klusterlet-addon-workmgr-7dfc659dd8-tx9rv\" (UID: \"4631961e-5c67-4a93-a16f-fc750132a627\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.319133 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.318960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9rrs\" (UniqueName: \"kubernetes.io/projected/04e79d99-63a6-4620-9b30-455f7e1b63fe-kube-api-access-d9rrs\") pod \"cluster-proxy-proxy-agent-85b6bdcd67-qc82s\" (UID: \"04e79d99-63a6-4620-9b30-455f7e1b63fe\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.460532 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.460472    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"
Sep 12 01:46:02.466217 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.466200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:02.471630 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.471611    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"
Sep 12 01:46:02.728340 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.728268    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h"]
Sep 12 01:46:02.730851 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:02.730820    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2caef38d_9c72_4104_9048_971011881e28.slice/crio-51bae0fc2ca928c7ff3a9f75928e3febfe0e1f2f32da5045263218d65bd04a61 WatchSource:0}: Error finding container 51bae0fc2ca928c7ff3a9f75928e3febfe0e1f2f32da5045263218d65bd04a61: Status 404 returned error can't find the container with id 51bae0fc2ca928c7ff3a9f75928e3febfe0e1f2f32da5045263218d65bd04a61
Sep 12 01:46:02.734352 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.734273    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"]
Sep 12 01:46:02.736703 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:02.736680    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4631961e_5c67_4a93_a16f_fc750132a627.slice/crio-bab656fcefc2f7583f8b70f4c8a97132217556242eb7e2f5d61e1586f0f71640 WatchSource:0}: Error finding container bab656fcefc2f7583f8b70f4c8a97132217556242eb7e2f5d61e1586f0f71640: Status 404 returned error can't find the container with id bab656fcefc2f7583f8b70f4c8a97132217556242eb7e2f5d61e1586f0f71640
Sep 12 01:46:02.751499 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.751478    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s"]
Sep 12 01:46:02.753749 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:02.753729    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e79d99_63a6_4620_9b30_455f7e1b63fe.slice/crio-c865ec5f48d2f21aafa95c9afbe27d86b4ec7162d7753dda450d62fcaf37d83e WatchSource:0}: Error finding container c865ec5f48d2f21aafa95c9afbe27d86b4ec7162d7753dda450d62fcaf37d83e: Status 404 returned error can't find the container with id c865ec5f48d2f21aafa95c9afbe27d86b4ec7162d7753dda450d62fcaf37d83e
Sep 12 01:46:02.941673 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.941643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv" event={"ID":"4631961e-5c67-4a93-a16f-fc750132a627","Type":"ContainerStarted","Data":"bab656fcefc2f7583f8b70f4c8a97132217556242eb7e2f5d61e1586f0f71640"}
Sep 12 01:46:02.942907 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.942882    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h" event={"ID":"2caef38d-9c72-4104-9048-971011881e28","Type":"ContainerStarted","Data":"51bae0fc2ca928c7ff3a9f75928e3febfe0e1f2f32da5045263218d65bd04a61"}
Sep 12 01:46:02.944198 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:02.944177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s" event={"ID":"04e79d99-63a6-4620-9b30-455f7e1b63fe","Type":"ContainerStarted","Data":"c865ec5f48d2f21aafa95c9afbe27d86b4ec7162d7753dda450d62fcaf37d83e"}
Sep 12 01:46:07.855090 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:07.855056    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2spkx"
Sep 12 01:46:07.965560 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:07.965530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h" event={"ID":"2caef38d-9c72-4104-9048-971011881e28","Type":"ContainerStarted","Data":"09a731e3274b76882af17f8f3b84e27a32b789c1f607d829aa2848eec54b6fa1"}
Sep 12 01:46:08.973367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:08.973329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s" event={"ID":"04e79d99-63a6-4620-9b30-455f7e1b63fe","Type":"ContainerStarted","Data":"5774edc684907d61297cab06c110a21d4180add6ca4b9fb94eddbc87ef3f042a"}
Sep 12 01:46:08.973755 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:08.973386    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s" event={"ID":"04e79d99-63a6-4620-9b30-455f7e1b63fe","Type":"ContainerStarted","Data":"9fbb307b781e5203e184f4192ece5a18501f1b074114db6963129b8804731b3a"}
Sep 12 01:46:08.973755 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:08.973406    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s" event={"ID":"04e79d99-63a6-4620-9b30-455f7e1b63fe","Type":"ContainerStarted","Data":"b7209fed917ff2333293dd3f47dc1975f2c839406bc8d4b51124c51447f6a7a3"}
Sep 12 01:46:08.994309 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:08.991558    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-85b6bdcd67-qc82s" podStartSLOduration=1.843986795 podStartE2EDuration="6.991537362s" totalImagesPullingTime="5.147550567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:08.989937025 +0000 UTC m=+62.916882915" watchObservedRunningTime="2026-09-12 01:46:08.991537362 +0000 UTC m=+62.918483249"
Sep 12 01:46:09.006666 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.006619    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-67b578c6fb-gp92h" podStartSLOduration=1.848966629 podStartE2EDuration="7.006601246s" totalImagesPullingTime="5.157634617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:09.006336797 +0000 UTC m=+62.933282684" watchObservedRunningTime="2026-09-12 01:46:09.006601246 +0000 UTC m=+62.933547130"
Sep 12 01:46:09.933965 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.933937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n8kws"
Sep 12 01:46:09.978873 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.978837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv" event={"ID":"4631961e-5c67-4a93-a16f-fc750132a627","Type":"ContainerStarted","Data":"20e79d152bb8f71089e7fef81ae24e5f734cf88a3796fc3a74aa70b62cd710eb"}
Sep 12 01:46:09.979345 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.979022    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:09.980633 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.980611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv"
Sep 12 01:46:09.994452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:09.994412    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dfc659dd8-tx9rv" podStartSLOduration=1.364014072 podStartE2EDuration="7.994398871s" totalImagesPullingTime="6.630384799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:46:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:09.993827299 +0000 UTC m=+63.920773187" watchObservedRunningTime="2026-09-12 01:46:09.994398871 +0000 UTC m=+63.921344790"
Sep 12 01:46:12.290391 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.290354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:46:12.290754 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.290402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:46:12.293006 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.292988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 12 01:46:12.293837 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.293819    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 12 01:46:12.303257 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.303234    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7fff888-e442-4677-a3c9-9b5d490bb332-original-pull-secret\") pod \"global-pull-secret-syncer-dzpgh\" (UID: \"d7fff888-e442-4677-a3c9-9b5d490bb332\") " pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:46:12.303257 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.303244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4db996df-7bb9-42a4-9c57-497e46831b9e-metrics-certs\") pod \"network-metrics-daemon-xk2nm\" (UID: \"4db996df-7bb9-42a4-9c57-497e46831b9e\") " pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:46:12.467416 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.467389    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dzpgh"
Sep 12 01:46:12.491718 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.491693    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:46:12.494482 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.494464    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 12 01:46:12.496883 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.496869    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fqtlc\""
Sep 12 01:46:12.504698 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.504680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xk2nm"
Sep 12 01:46:12.504796 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.504787    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 12 01:46:12.515597 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.515578    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btc88\" (UniqueName: \"kubernetes.io/projected/db1bbe8c-18c3-48f1-992c-282a06e7f413-kube-api-access-btc88\") pod \"network-check-target-pb84r\" (UID: \"db1bbe8c-18c3-48f1-992c-282a06e7f413\") " pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:46:12.762479 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.762448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dzpgh"]
Sep 12 01:46:12.765673 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:12.765647    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7fff888_e442_4677_a3c9_9b5d490bb332.slice/crio-b03bbfc50f033f9e1a45403575b307b4bf10e16ab108b92893dbcc881872b4c5 WatchSource:0}: Error finding container b03bbfc50f033f9e1a45403575b307b4bf10e16ab108b92893dbcc881872b4c5: Status 404 returned error can't find the container with id b03bbfc50f033f9e1a45403575b307b4bf10e16ab108b92893dbcc881872b4c5
Sep 12 01:46:12.801919 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.801870    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmzff\""
Sep 12 01:46:12.808507 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.808481    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xk2nm"]
Sep 12 01:46:12.810249 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.810231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:46:12.811441 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:12.811421    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db996df_7bb9_42a4_9c57_497e46831b9e.slice/crio-f8ac8ce555211f0ee43156f15f8aac1f51644ca3539c274753ee72f973a8c0a6 WatchSource:0}: Error finding container f8ac8ce555211f0ee43156f15f8aac1f51644ca3539c274753ee72f973a8c0a6: Status 404 returned error can't find the container with id f8ac8ce555211f0ee43156f15f8aac1f51644ca3539c274753ee72f973a8c0a6
Sep 12 01:46:12.989282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.989249    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2nm" event={"ID":"4db996df-7bb9-42a4-9c57-497e46831b9e","Type":"ContainerStarted","Data":"f8ac8ce555211f0ee43156f15f8aac1f51644ca3539c274753ee72f973a8c0a6"}
Sep 12 01:46:12.990422 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:12.990394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzpgh" event={"ID":"d7fff888-e442-4677-a3c9-9b5d490bb332","Type":"ContainerStarted","Data":"b03bbfc50f033f9e1a45403575b307b4bf10e16ab108b92893dbcc881872b4c5"}
Sep 12 01:46:13.099173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:13.099119    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pb84r"]
Sep 12 01:46:13.102024 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:13.101989    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1bbe8c_18c3_48f1_992c_282a06e7f413.slice/crio-7b6c433ccea31f55f4c2dfd32363ba410cbb215dedf5eaa8e7392937e8d61d7b WatchSource:0}: Error finding container 7b6c433ccea31f55f4c2dfd32363ba410cbb215dedf5eaa8e7392937e8d61d7b: Status 404 returned error can't find the container with id 7b6c433ccea31f55f4c2dfd32363ba410cbb215dedf5eaa8e7392937e8d61d7b
Sep 12 01:46:13.995650 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:13.995607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb84r" event={"ID":"db1bbe8c-18c3-48f1-992c-282a06e7f413","Type":"ContainerStarted","Data":"7b6c433ccea31f55f4c2dfd32363ba410cbb215dedf5eaa8e7392937e8d61d7b"}
Sep 12 01:46:15.002211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:15.002160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2nm" event={"ID":"4db996df-7bb9-42a4-9c57-497e46831b9e","Type":"ContainerStarted","Data":"c0f18d1e615a06090ee80858eac8551818c569e1e7148cbcee7cfe7da650ef01"}
Sep 12 01:46:15.002698 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:15.002230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xk2nm" event={"ID":"4db996df-7bb9-42a4-9c57-497e46831b9e","Type":"ContainerStarted","Data":"846e5fbf0a08f24bffdc7962669264749ca7862c81e2daa829f075a213ec3409"}
Sep 12 01:46:15.017289 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:15.017235    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xk2nm" podStartSLOduration=67.564758954 podStartE2EDuration="1m9.017215316s" totalImagesPullingTime="1.452456362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:15.01639843 +0000 UTC m=+68.943344318" watchObservedRunningTime="2026-09-12 01:46:15.017215316 +0000 UTC m=+68.944161198"
Sep 12 01:46:16.947875 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.947805    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pmssx"]
Sep 12 01:46:16.951244 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.951169    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:16.953636 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.953601    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 12 01:46:16.953776 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.953600    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 12 01:46:16.953917 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.953893    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 12 01:46:16.954101 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.954082    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 12 01:46:16.954207 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.954123    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xdtzd\""
Sep 12 01:46:16.954643 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.954618    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 12 01:46:16.954757 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:16.954675    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 12 01:46:17.025741 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.025903 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-metrics-client-ca\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.025903 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7cq\" (UniqueName: \"kubernetes.io/projected/f5ac8580-c05e-4412-8f9e-163eed7c1eac-kube-api-access-kd7cq\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.025903 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025869    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-sys\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.026099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-wtmp\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.026099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.025934    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-textfile\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.026099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.026011    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.026099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.026090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-tls\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.026262 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.026127    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-root\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126783 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-metrics-client-ca\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126783 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126796    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7cq\" (UniqueName: \"kubernetes.io/projected/f5ac8580-c05e-4412-8f9e-163eed7c1eac-kube-api-access-kd7cq\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126831    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-sys\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-wtmp\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126896    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-textfile\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.126998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.126956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-sys\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.127498 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.127469    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-tls\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.127623 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.127500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-textfile\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.127623 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.127565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-root\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.127738 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.127632    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.128341 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.128288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-root\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.128874 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.128786    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-metrics-client-ca\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.128984 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.128728    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-accelerators-collector-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.128984 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.127569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-wtmp\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.132848 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.132815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-tls\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.133432 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.133400    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5ac8580-c05e-4412-8f9e-163eed7c1eac-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.141303 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.141284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7cq\" (UniqueName: \"kubernetes.io/projected/f5ac8580-c05e-4412-8f9e-163eed7c1eac-kube-api-access-kd7cq\") pod \"node-exporter-pmssx\" (UID: \"f5ac8580-c05e-4412-8f9e-163eed7c1eac\") " pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.263462 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:17.263375    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pmssx"
Sep 12 01:46:17.656481 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:46:17.656441    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ac8580_c05e_4412_8f9e_163eed7c1eac.slice/crio-d850d90a5c856feffc899a538fac8b9f03b7c1ed9f0b9064346b2f339b7823a7 WatchSource:0}: Error finding container d850d90a5c856feffc899a538fac8b9f03b7c1ed9f0b9064346b2f339b7823a7: Status 404 returned error can't find the container with id d850d90a5c856feffc899a538fac8b9f03b7c1ed9f0b9064346b2f339b7823a7
Sep 12 01:46:18.016922 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.016828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dzpgh" event={"ID":"d7fff888-e442-4677-a3c9-9b5d490bb332","Type":"ContainerStarted","Data":"76f97b5e75cfde54ccabeb2d1347b1254359e3a5933d602b84f9099c643257f4"}
Sep 12 01:46:18.019193 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.019163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pb84r" event={"ID":"db1bbe8c-18c3-48f1-992c-282a06e7f413","Type":"ContainerStarted","Data":"a5d2eb10eea1fd9be2caad13810a53c934b616513f5b67fe70e410f978f4cacc"}
Sep 12 01:46:18.019338 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.019264    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:46:18.020920 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.020900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmssx" event={"ID":"f5ac8580-c05e-4412-8f9e-163eed7c1eac","Type":"ContainerStarted","Data":"d850d90a5c856feffc899a538fac8b9f03b7c1ed9f0b9064346b2f339b7823a7"}
Sep 12 01:46:18.036080 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.036013    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dzpgh" podStartSLOduration=67.141170371 podStartE2EDuration="1m12.03599836s" totalImagesPullingTime="4.894827989s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:18.034759816 +0000 UTC m=+71.961705703" watchObservedRunningTime="2026-09-12 01:46:18.03599836 +0000 UTC m=+71.962944245"
Sep 12 01:46:18.053364 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:18.053328    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pb84r" podStartSLOduration=67.4907215 podStartE2EDuration="1m12.053316307s" totalImagesPullingTime="4.562594807s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:45:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:18.052235777 +0000 UTC m=+71.979181665" watchObservedRunningTime="2026-09-12 01:46:18.053316307 +0000 UTC m=+71.980262193"
Sep 12 01:46:19.025969 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:19.025937    2582 generic.go:356] "Generic (PLEG): container finished" podID="f5ac8580-c05e-4412-8f9e-163eed7c1eac" containerID="abbc989fbc43b8f6eb2e6257795e54c4abb0f8473562a5e380e8a1ea1a48d485" exitCode=0
Sep 12 01:46:19.026450 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:19.026022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmssx" event={"ID":"f5ac8580-c05e-4412-8f9e-163eed7c1eac","Type":"ContainerDied","Data":"abbc989fbc43b8f6eb2e6257795e54c4abb0f8473562a5e380e8a1ea1a48d485"}
Sep 12 01:46:20.031487 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:20.031450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmssx" event={"ID":"f5ac8580-c05e-4412-8f9e-163eed7c1eac","Type":"ContainerStarted","Data":"c9ddba42654be158087825d2f10814c0023b9c3d700677007c1848bedd77209c"}
Sep 12 01:46:20.031936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:20.031502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pmssx" event={"ID":"f5ac8580-c05e-4412-8f9e-163eed7c1eac","Type":"ContainerStarted","Data":"eff37b23df2becde18ce6c17aa9fc2b8182c0054af94510727fd46bdad28c42b"}
Sep 12 01:46:20.049773 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:20.049721    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pmssx" podStartSLOduration=3.207421553 podStartE2EDuration="4.049706165s" totalImagesPullingTime="780.442063ms" totalInitContainerRuntime="61.842549ms" isStatefulPod=true podCreationTimestamp="2026-09-12 01:46:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:46:20.048771003 +0000 UTC m=+73.975716889" watchObservedRunningTime="2026-09-12 01:46:20.049706165 +0000 UTC m=+73.976652045"
Sep 12 01:46:49.028998 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:46:49.028967    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pb84r"
Sep 12 01:50:06.472391 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:06.472363    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:50:06.477148 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:06.477126    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 12 01:50:06.477267 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:06.477204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:50:25.072627 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.072589    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"]
Sep 12 01:50:25.074509 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.074491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.076947 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.076925    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 12 01:50:25.077870 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.077848    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-86lxz\""
Sep 12 01:50:25.077965 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.077867    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 12 01:50:25.084138 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.084117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"]
Sep 12 01:50:25.112202 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.112180    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ns7j\" (UniqueName: \"kubernetes.io/projected/5b196256-5b66-492a-bf34-4cd859982789-kube-api-access-2ns7j\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.112314 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.112217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b196256-5b66-492a-bf34-4cd859982789-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.213313 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.213289    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ns7j\" (UniqueName: \"kubernetes.io/projected/5b196256-5b66-492a-bf34-4cd859982789-kube-api-access-2ns7j\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.213395 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.213328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b196256-5b66-492a-bf34-4cd859982789-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.213654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.213638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b196256-5b66-492a-bf34-4cd859982789-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.225572 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.225549    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ns7j\" (UniqueName: \"kubernetes.io/projected/5b196256-5b66-492a-bf34-4cd859982789-kube-api-access-2ns7j\") pod \"openshift-lws-operator-fd8ccff4c-xjwpd\" (UID: \"5b196256-5b66-492a-bf34-4cd859982789\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.383549 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.383531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"
Sep 12 01:50:25.647174 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.647110    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd"]
Sep 12 01:50:25.650135 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:50:25.650108    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b196256_5b66_492a_bf34_4cd859982789.slice/crio-008525831df950a71a352e0377466e3c4f97d4769ab3ddf44c6894862f86014d WatchSource:0}: Error finding container 008525831df950a71a352e0377466e3c4f97d4769ab3ddf44c6894862f86014d: Status 404 returned error can't find the container with id 008525831df950a71a352e0377466e3c4f97d4769ab3ddf44c6894862f86014d
Sep 12 01:50:25.651572 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.651555    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:50:25.803910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:25.803883    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd" event={"ID":"5b196256-5b66-492a-bf34-4cd859982789","Type":"ContainerStarted","Data":"008525831df950a71a352e0377466e3c4f97d4769ab3ddf44c6894862f86014d"}
Sep 12 01:50:28.816644 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:28.816601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd" event={"ID":"5b196256-5b66-492a-bf34-4cd859982789","Type":"ContainerStarted","Data":"5e3cc12fa04ac5e39e1fdcdeb37114be096b4c00859a5fb6a81ea4fa9e53c9de"}
Sep 12 01:50:28.833506 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:28.833452    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xjwpd" podStartSLOduration=1.402759479 podStartE2EDuration="3.833432034s" totalImagesPullingTime="2.430672555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:50:28.832277152 +0000 UTC m=+322.759223039" watchObservedRunningTime="2026-09-12 01:50:28.833432034 +0000 UTC m=+322.760377921"
Sep 12 01:50:44.989955 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.989917    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"]
Sep 12 01:50:44.996224 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.996199    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:44.999211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.999189    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 12 01:50:44.999316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.999232    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 12 01:50:44.999749 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.999733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qhcs5\""
Sep 12 01:50:44.999827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:44.999733    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 12 01:50:45.000052 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.000019    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 12 01:50:45.011638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.011617    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"]
Sep 12 01:50:45.043440 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.043413    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.043528 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.043448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.043528 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.043476    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.043596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.043552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.043596 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.043575    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj74j\" (UniqueName: \"kubernetes.io/projected/4549777b-9184-4dd4-8f85-787ad4280243-kube-api-access-fj74j\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.144750 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.144728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.144845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.144759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj74j\" (UniqueName: \"kubernetes.io/projected/4549777b-9184-4dd4-8f85-787ad4280243-kube-api-access-fj74j\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.144845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.144787    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.144845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.144804    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.144845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.144833    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.145214 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.145191    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-manifests\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.147072 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.147032    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4549777b-9184-4dd4-8f85-787ad4280243-tmp\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.147332 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.147313    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-apiservice-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.147395 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.147312    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4549777b-9184-4dd4-8f85-787ad4280243-webhook-cert\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.152910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.152890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj74j\" (UniqueName: \"kubernetes.io/projected/4549777b-9184-4dd4-8f85-787ad4280243-kube-api-access-fj74j\") pod \"opendatahub-operator-controller-manager-578c78994-6mdxq\" (UID: \"4549777b-9184-4dd4-8f85-787ad4280243\") " pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.305606 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.305545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:45.592456 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.592386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"]
Sep 12 01:50:45.595265 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:50:45.595238    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4549777b_9184_4dd4_8f85_787ad4280243.slice/crio-682be6810b9511676ce874c7e1a1bde461d9009dfa72bf6047b272885305d0f5 WatchSource:0}: Error finding container 682be6810b9511676ce874c7e1a1bde461d9009dfa72bf6047b272885305d0f5: Status 404 returned error can't find the container with id 682be6810b9511676ce874c7e1a1bde461d9009dfa72bf6047b272885305d0f5
Sep 12 01:50:45.874379 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:45.874344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq" event={"ID":"4549777b-9184-4dd4-8f85-787ad4280243","Type":"ContainerStarted","Data":"682be6810b9511676ce874c7e1a1bde461d9009dfa72bf6047b272885305d0f5"}
Sep 12 01:50:48.887672 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:48.887635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq" event={"ID":"4549777b-9184-4dd4-8f85-787ad4280243","Type":"ContainerStarted","Data":"9e7fbae865d8d9900e9eb5018772eec8a4ccdfddd0a35c667835cccb93b9228a"}
Sep 12 01:50:49.895689 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:49.895656    2582 generic.go:356] "Generic (PLEG): container finished" podID="4549777b-9184-4dd4-8f85-787ad4280243" containerID="9e7fbae865d8d9900e9eb5018772eec8a4ccdfddd0a35c667835cccb93b9228a" exitCode=0
Sep 12 01:50:49.896128 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:49.895724    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq" event={"ID":"4549777b-9184-4dd4-8f85-787ad4280243","Type":"ContainerDied","Data":"9e7fbae865d8d9900e9eb5018772eec8a4ccdfddd0a35c667835cccb93b9228a"}
Sep 12 01:50:50.901480 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:50.901439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq" event={"ID":"4549777b-9184-4dd4-8f85-787ad4280243","Type":"ContainerStarted","Data":"ca4b03f7a8dc7b91f2869841f5e5aee7551518fd410085802c6ef71eca22868d"}
Sep 12 01:50:50.902024 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:50.901564    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:50:50.924827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:50:50.924780    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq" podStartSLOduration=2.423292909 podStartE2EDuration="6.924765466s" totalImagesPullingTime="3.429331861s" totalInitContainerRuntime="1.072140696s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:50:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:50:50.924087824 +0000 UTC m=+344.851033710" watchObservedRunningTime="2026-09-12 01:50:50.924765466 +0000 UTC m=+344.851711352"
Sep 12 01:51:01.907948 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:01.907869    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-578c78994-6mdxq"
Sep 12 01:51:05.433559 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.433523    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"]
Sep 12 01:51:05.436252 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.436232    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.438669 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.438649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 12 01:51:05.438779 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.438689    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 01:51:05.439965 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.439949    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 12 01:51:05.440073 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.440021    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-q5vdz\""
Sep 12 01:51:05.440143 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.440080    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 12 01:51:05.453146 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.453125    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"]
Sep 12 01:51:05.489526 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.489502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38422c3e-d557-4476-a945-4d6b9bb5f942-tmp\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.489622 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.489540    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrt8n\" (UniqueName: \"kubernetes.io/projected/38422c3e-d557-4476-a945-4d6b9bb5f942-kube-api-access-xrt8n\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.489622 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.489609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38422c3e-d557-4476-a945-4d6b9bb5f942-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.590842 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.590806    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38422c3e-d557-4476-a945-4d6b9bb5f942-tmp\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.590950 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.590856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrt8n\" (UniqueName: \"kubernetes.io/projected/38422c3e-d557-4476-a945-4d6b9bb5f942-kube-api-access-xrt8n\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.590950 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.590880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38422c3e-d557-4476-a945-4d6b9bb5f942-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.592994 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.592974    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38422c3e-d557-4476-a945-4d6b9bb5f942-tmp\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.593129 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.593112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38422c3e-d557-4476-a945-4d6b9bb5f942-tls-certs\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.599386 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.599367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrt8n\" (UniqueName: \"kubernetes.io/projected/38422c3e-d557-4476-a945-4d6b9bb5f942-kube-api-access-xrt8n\") pod \"kube-auth-proxy-69c99bdc57-4v2nr\" (UID: \"38422c3e-d557-4476-a945-4d6b9bb5f942\") " pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:05.745791 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:05.745724    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"
Sep 12 01:51:06.015492 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.015468    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr"]
Sep 12 01:51:06.017454 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:06.017432    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38422c3e_d557_4476_a945_4d6b9bb5f942.slice/crio-aacd7082a8e37096622d32f410729d14b2dfa8b0dec7c322c58007ee544c8ef9 WatchSource:0}: Error finding container aacd7082a8e37096622d32f410729d14b2dfa8b0dec7c322c58007ee544c8ef9: Status 404 returned error can't find the container with id aacd7082a8e37096622d32f410729d14b2dfa8b0dec7c322c58007ee544c8ef9
Sep 12 01:51:06.678935 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.678902    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:06.682211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.682186    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.685177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.684901    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 12 01:51:06.685177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.684914    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 12 01:51:06.685177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.684909    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 12 01:51:06.686148 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.686109    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 12 01:51:06.686332 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.686184    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-drt6b\""
Sep 12 01:51:06.697330 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.697287    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:06.800890 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.800853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.800900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.800925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.800955    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.801002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.801067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b8n2\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.801359 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.801115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902361 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b8n2\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902509    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.902796 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.902540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.903397 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.903361    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.905547 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.905522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.905655 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.905532    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.905760 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.905739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.906276 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.906250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.912275 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.912230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b8n2\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.912389 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.912286    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6sb9v\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:06.959700 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.959636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr" event={"ID":"38422c3e-d557-4476-a945-4d6b9bb5f942","Type":"ContainerStarted","Data":"aacd7082a8e37096622d32f410729d14b2dfa8b0dec7c322c58007ee544c8ef9"}
Sep 12 01:51:06.993857 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:06.993834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:07.278331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:07.278255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:07.281671 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:07.281638    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc89adfc_f8ef_4253_9231_64416aafebaa.slice/crio-819b19a9a4fd8bdf75b40cfdd8e20beed4a04b3ba02c1c7498cb2540c2ad806f WatchSource:0}: Error finding container 819b19a9a4fd8bdf75b40cfdd8e20beed4a04b3ba02c1c7498cb2540c2ad806f: Status 404 returned error can't find the container with id 819b19a9a4fd8bdf75b40cfdd8e20beed4a04b3ba02c1c7498cb2540c2ad806f
Sep 12 01:51:07.965341 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:07.965299    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" event={"ID":"cc89adfc-f8ef-4253-9231-64416aafebaa","Type":"ContainerStarted","Data":"819b19a9a4fd8bdf75b40cfdd8e20beed4a04b3ba02c1c7498cb2540c2ad806f"}
Sep 12 01:51:08.730099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:08.730075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 12 01:51:08.971607 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:08.971574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr" event={"ID":"38422c3e-d557-4476-a945-4d6b9bb5f942","Type":"ContainerStarted","Data":"5c317ee7220fc7d6cc326fe83fa6a898a1a75eba1cae67f9009be80d4a27b76a"}
Sep 12 01:51:08.988241 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:08.988148    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69c99bdc57-4v2nr" podStartSLOduration=1.279816851 podStartE2EDuration="3.988127231s" totalImagesPullingTime="2.70831038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:08.987359041 +0000 UTC m=+362.914304939" watchObservedRunningTime="2026-09-12 01:51:08.988127231 +0000 UTC m=+362.915073118"
Sep 12 01:51:10.567160 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:10.567096    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:51:10.567532 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:10.567194    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:51:10.983219 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:10.983186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" event={"ID":"cc89adfc-f8ef-4253-9231-64416aafebaa","Type":"ContainerStarted","Data":"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"}
Sep 12 01:51:10.983385 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:10.983329    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:11.002612 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:11.002562    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" podStartSLOduration=1.720048835 podStartE2EDuration="5.002548732s" totalImagesPullingTime="3.282499897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:11.001343416 +0000 UTC m=+364.928289300" watchObservedRunningTime="2026-09-12 01:51:11.002548732 +0000 UTC m=+364.929494618"
Sep 12 01:51:11.988817 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:11.988783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:16.119807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.119775    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r57lk"]
Sep 12 01:51:16.122783 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.122762    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.125135 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.125114    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 12 01:51:16.125243 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.125146    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5jg7p\""
Sep 12 01:51:16.132395 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.132359    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r57lk"]
Sep 12 01:51:16.273858 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.273826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2mf9\" (UniqueName: \"kubernetes.io/projected/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-kube-api-access-d2mf9\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.274051 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.273885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.375135 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.375054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2mf9\" (UniqueName: \"kubernetes.io/projected/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-kube-api-access-d2mf9\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.375135 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.375126    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.375334 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:51:16.375220    2582 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 12 01:51:16.375334 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:51:16.375271    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert podName:37e490e7-2fe5-4c9c-a74d-7df7389d7ff7 nodeName:}" failed. No retries permitted until 2026-09-12 01:51:16.875256063 +0000 UTC m=+370.802201939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert") pod "odh-model-controller-7998c67dc4-r57lk" (UID: "37e490e7-2fe5-4c9c-a74d-7df7389d7ff7") : secret "odh-model-controller-webhook-cert" not found
Sep 12 01:51:16.384197 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.384166    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2mf9\" (UniqueName: \"kubernetes.io/projected/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-kube-api-access-d2mf9\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.531000 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.530969    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6k8q"]
Sep 12 01:51:16.533855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.533837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.536228 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.536198    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 12 01:51:16.536342 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.536297    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-7wznw\""
Sep 12 01:51:16.544518 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.544499    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6k8q"]
Sep 12 01:51:16.677885 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.677791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jj9\" (UniqueName: \"kubernetes.io/projected/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-kube-api-access-g7jj9\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.677885 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.677843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.779117 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.779083    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jj9\" (UniqueName: \"kubernetes.io/projected/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-kube-api-access-g7jj9\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.779117 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.779121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.781557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.781538    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-tls-certs\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.787618 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.787598    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jj9\" (UniqueName: \"kubernetes.io/projected/c1ac51ac-faec-41be-b0b9-a09e6b0d1c11-kube-api-access-g7jj9\") pod \"model-serving-api-54d89bd5fc-q6k8q\" (UID: \"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11\") " pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.844571 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.844546    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:16.879741 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.879704    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:16.882859 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:16.882831    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/37e490e7-2fe5-4c9c-a74d-7df7389d7ff7-cert\") pod \"odh-model-controller-7998c67dc4-r57lk\" (UID: \"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7\") " pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:17.032553 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:17.032470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:17.146120 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:17.146093    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-q6k8q"]
Sep 12 01:51:17.149666 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:17.149638    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1ac51ac_faec_41be_b0b9_a09e6b0d1c11.slice/crio-1585431e49d28732cc4d818b48546e22f2099fc42ef4616fbc90a2fcce1a4e5a WatchSource:0}: Error finding container 1585431e49d28732cc4d818b48546e22f2099fc42ef4616fbc90a2fcce1a4e5a: Status 404 returned error can't find the container with id 1585431e49d28732cc4d818b48546e22f2099fc42ef4616fbc90a2fcce1a4e5a
Sep 12 01:51:17.301936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:17.301860    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-r57lk"]
Sep 12 01:51:17.304504 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:17.304477    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e490e7_2fe5_4c9c_a74d_7df7389d7ff7.slice/crio-c7079d52bd3e50c86341f0809f275dc18c2903286981d84393312ce9df28238b WatchSource:0}: Error finding container c7079d52bd3e50c86341f0809f275dc18c2903286981d84393312ce9df28238b: Status 404 returned error can't find the container with id c7079d52bd3e50c86341f0809f275dc18c2903286981d84393312ce9df28238b
Sep 12 01:51:18.010567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:18.010521    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q" event={"ID":"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11","Type":"ContainerStarted","Data":"1585431e49d28732cc4d818b48546e22f2099fc42ef4616fbc90a2fcce1a4e5a"}
Sep 12 01:51:18.012023 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:18.011992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk" event={"ID":"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7","Type":"ContainerStarted","Data":"c7079d52bd3e50c86341f0809f275dc18c2903286981d84393312ce9df28238b"}
Sep 12 01:51:21.026069 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:21.026004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk" event={"ID":"37e490e7-2fe5-4c9c-a74d-7df7389d7ff7","Type":"ContainerStarted","Data":"8f50c257d1941ca42a7d5997b0b176cf33dd5df71987b2839b1dc17a002e955d"}
Sep 12 01:51:21.026486 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:21.026096    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:51:21.027517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:21.027490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q" event={"ID":"c1ac51ac-faec-41be-b0b9-a09e6b0d1c11","Type":"ContainerStarted","Data":"3a51c715d1274a35812b848e610aa23d3faa913ad3e0f7346360ba77ab0e8e86"}
Sep 12 01:51:21.027641 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:21.027552    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:21.046328 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:21.046284    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk" podStartSLOduration=2.047117054 podStartE2EDuration="5.046269494s" totalImagesPullingTime="2.99915244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:21.045131111 +0000 UTC m=+374.972076998" watchObservedRunningTime="2026-09-12 01:51:21.046269494 +0000 UTC m=+374.973215380"
Sep 12 01:51:23.028999 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.028942    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q" podStartSLOduration=3.921752395 podStartE2EDuration="7.028920302s" totalImagesPullingTime="3.107167907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:21.060610896 +0000 UTC m=+374.987556786" watchObservedRunningTime="2026-09-12 01:51:23.028920302 +0000 UTC m=+376.955866193"
Sep 12 01:51:23.030009 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.029991    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"]
Sep 12 01:51:23.035102 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.035080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.045092 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.045069    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"]
Sep 12 01:51:23.129601 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129562    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129734 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6sf\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-kube-api-access-pn6sf\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129734 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129825 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129734    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129825 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129892 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.129892 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.129882    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/413ca503-d97d-4d54-9883-17eef7612b9e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230274 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230274 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230300    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/413ca503-d97d-4d54-9883-17eef7612b9e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230348    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6sf\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-kube-api-access-pn6sf\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230374    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.230483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.230405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.231227 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.231197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.232679 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.232648    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/413ca503-d97d-4d54-9883-17eef7612b9e-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.232877 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.232857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.232920 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.232889    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.232958 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.232948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/413ca503-d97d-4d54-9883-17eef7612b9e-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.246675 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.246655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.247023 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.247003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6sf\" (UniqueName: \"kubernetes.io/projected/413ca503-d97d-4d54-9883-17eef7612b9e-kube-api-access-pn6sf\") pod \"istiod-openshift-gateway-5677ccb49b-djtnb\" (UID: \"413ca503-d97d-4d54-9883-17eef7612b9e\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.344813 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.344741    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:23.675780 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.675755    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"]
Sep 12 01:51:23.677892 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:23.677868    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413ca503_d97d_4d54_9883_17eef7612b9e.slice/crio-930d64f3e88d68b9dbc12ed4ecc962a3aff55a0107ad99f78977a4092be2d9e8 WatchSource:0}: Error finding container 930d64f3e88d68b9dbc12ed4ecc962a3aff55a0107ad99f78977a4092be2d9e8: Status 404 returned error can't find the container with id 930d64f3e88d68b9dbc12ed4ecc962a3aff55a0107ad99f78977a4092be2d9e8
Sep 12 01:51:23.680963 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.680925    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:51:23.681033 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:23.681004    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:51:24.040257 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.040169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb" event={"ID":"413ca503-d97d-4d54-9883-17eef7612b9e","Type":"ContainerStarted","Data":"5025842db95149b692e39b750e2e6df5b3d6d3198e7d93dd16a1b1d3fb5aba6c"}
Sep 12 01:51:24.040257 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.040224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb" event={"ID":"413ca503-d97d-4d54-9883-17eef7612b9e","Type":"ContainerStarted","Data":"930d64f3e88d68b9dbc12ed4ecc962a3aff55a0107ad99f78977a4092be2d9e8"}
Sep 12 01:51:24.040730 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.040399    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:24.041640 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.041616    2582 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-djtnb container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 01:51:24.041747 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.041663    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb" podUID="413ca503-d97d-4d54-9883-17eef7612b9e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 01:51:24.061262 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:24.061216    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb" podStartSLOduration=1.061202848 podStartE2EDuration="1.061202848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:24.059610146 +0000 UTC m=+377.986556082" watchObservedRunningTime="2026-09-12 01:51:24.061202848 +0000 UTC m=+377.988148733"
Sep 12 01:51:25.044370 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.044337    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-djtnb"
Sep 12 01:51:25.104465 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.104431    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:25.104679 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.104635    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" podUID="cc89adfc-f8ef-4253-9231-64416aafebaa" containerName="discovery" containerID="cri-o://e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848" gracePeriod=30
Sep 12 01:51:25.346810 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.346787    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:25.446482 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446452    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446637 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446637 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446513    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446637 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4b8n2\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446637 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446637 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446622    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446887 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446645    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert\") pod \"cc89adfc-f8ef-4253-9231-64416aafebaa\" (UID: \"cc89adfc-f8ef-4253-9231-64416aafebaa\") "
Sep 12 01:51:25.446949 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.446886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:51:25.448900 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.448870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:51:25.449029 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.448921    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts" (OuterVolumeSpecName: "cacerts") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:51:25.449200 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.449171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs" (OuterVolumeSpecName: "local-certs") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:51:25.449317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.449273    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:51:25.449317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.449293    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:51:25.449317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.449292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2" (OuterVolumeSpecName: "kube-api-access-4b8n2") pod "cc89adfc-f8ef-4253-9231-64416aafebaa" (UID: "cc89adfc-f8ef-4253-9231-64416aafebaa"). InnerVolumeSpecName "kube-api-access-4b8n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:51:25.547514 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547473    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-kubeconfig\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547514 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547509    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-dns-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547523    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/cc89adfc-f8ef-4253-9231-64416aafebaa-cacerts\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547536    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-csr-ca-configmap\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547550    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc89adfc-f8ef-4253-9231-64416aafebaa-local-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547561    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b8n2\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-kube-api-access-4b8n2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.547677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.547576    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc89adfc-f8ef-4253-9231-64416aafebaa-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:51:25.660933 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.660877    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"]
Sep 12 01:51:25.661177 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.661162    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc89adfc-f8ef-4253-9231-64416aafebaa" containerName="discovery"
Sep 12 01:51:25.661251 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.661178    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc89adfc-f8ef-4253-9231-64416aafebaa" containerName="discovery"
Sep 12 01:51:25.661251 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.661220    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc89adfc-f8ef-4253-9231-64416aafebaa" containerName="discovery"
Sep 12 01:51:25.663254 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.663235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.665968 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.665940    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 12 01:51:25.666390 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.666372    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-xr4hm\""
Sep 12 01:51:25.680010 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.679987    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"]
Sep 12 01:51:25.749872 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.749845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5551845c-0b0d-4b98-8f57-7372cef926ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.749984 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.749879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nw5r\" (UniqueName: \"kubernetes.io/projected/5551845c-0b0d-4b98-8f57-7372cef926ec-kube-api-access-2nw5r\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.850503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.850473    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5551845c-0b0d-4b98-8f57-7372cef926ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.850503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.850510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nw5r\" (UniqueName: \"kubernetes.io/projected/5551845c-0b0d-4b98-8f57-7372cef926ec-kube-api-access-2nw5r\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.853084 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.853055    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5551845c-0b0d-4b98-8f57-7372cef926ec-cert\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.860544 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.860522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nw5r\" (UniqueName: \"kubernetes.io/projected/5551845c-0b0d-4b98-8f57-7372cef926ec-kube-api-access-2nw5r\") pod \"kserve-controller-manager-d6f4b65b8-cvq8m\" (UID: \"5551845c-0b0d-4b98-8f57-7372cef926ec\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:25.973376 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:25.973289    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:26.049100 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.049074    2582 generic.go:356] "Generic (PLEG): container finished" podID="cc89adfc-f8ef-4253-9231-64416aafebaa" containerID="e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848" exitCode=0
Sep 12 01:51:26.049414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.049140    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"
Sep 12 01:51:26.049414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.049152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" event={"ID":"cc89adfc-f8ef-4253-9231-64416aafebaa","Type":"ContainerDied","Data":"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"}
Sep 12 01:51:26.049414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.049204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v" event={"ID":"cc89adfc-f8ef-4253-9231-64416aafebaa","Type":"ContainerDied","Data":"819b19a9a4fd8bdf75b40cfdd8e20beed4a04b3ba02c1c7498cb2540c2ad806f"}
Sep 12 01:51:26.049414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.049227    2582 scope.go:117] "RemoveContainer" containerID="e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"
Sep 12 01:51:26.057898 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.057877    2582 scope.go:117] "RemoveContainer" containerID="e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"
Sep 12 01:51:26.058174 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:51:26.058155    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848\": container with ID starting with e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848 not found: ID does not exist" containerID="e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"
Sep 12 01:51:26.058247 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.058183    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848"} err="failed to get container status \"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848\": rpc error: code = NotFound desc = could not find container \"e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848\": container with ID starting with e3129b3564b61bb01a07518ee8b8629b2799835e94d5701aafa0d5d06e183848 not found: ID does not exist"
Sep 12 01:51:26.091650 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.091621    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:26.095563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.095542    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6sb9v"]
Sep 12 01:51:26.242234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.242154    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"]
Sep 12 01:51:26.244969 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:51:26.244942    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5551845c_0b0d_4b98_8f57_7372cef926ec.slice/crio-800bfcf42fb88580a975f74f15e6e761608c438b60ed92b4a94110d28f15575d WatchSource:0}: Error finding container 800bfcf42fb88580a975f74f15e6e761608c438b60ed92b4a94110d28f15575d: Status 404 returned error can't find the container with id 800bfcf42fb88580a975f74f15e6e761608c438b60ed92b4a94110d28f15575d
Sep 12 01:51:26.554201 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:26.554129    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc89adfc-f8ef-4253-9231-64416aafebaa" path="/var/lib/kubelet/pods/cc89adfc-f8ef-4253-9231-64416aafebaa/volumes"
Sep 12 01:51:27.056465 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:27.056426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m" event={"ID":"5551845c-0b0d-4b98-8f57-7372cef926ec","Type":"ContainerStarted","Data":"800bfcf42fb88580a975f74f15e6e761608c438b60ed92b4a94110d28f15575d"}
Sep 12 01:51:29.071013 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:29.070972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m" event={"ID":"5551845c-0b0d-4b98-8f57-7372cef926ec","Type":"ContainerStarted","Data":"2ec3718d3784a73f017ad83d8be4c5427b699a4f34f995382795e475ba79f77c"}
Sep 12 01:51:29.071394 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:29.071090    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:51:29.090178 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:29.090133    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m" podStartSLOduration=2.0681008419999998 podStartE2EDuration="4.090117072s" totalImagesPullingTime="2.02201623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:51:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:51:29.089960515 +0000 UTC m=+383.016906412" watchObservedRunningTime="2026-09-12 01:51:29.090117072 +0000 UTC m=+383.017062959"
Sep 12 01:51:32.035378 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:32.035351    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-q6k8q"
Sep 12 01:51:32.070472 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:51:32.070445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-r57lk"
Sep 12 01:52:00.080032 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:00.080002    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cvq8m"
Sep 12 01:52:45.731740 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.731704    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"]
Sep 12 01:52:45.738098 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.738067    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:45.740799 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.740779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lbd2b\""
Sep 12 01:52:45.741823 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.741804    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 01:52:45.741900 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.741808    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 01:52:45.752140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.752114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"]
Sep 12 01:52:45.801067 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.801022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5l5b\" (UniqueName: \"kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b\") pod \"limitador-operator-controller-manager-74495c69f8-g8l6h\" (UID: \"c773d4a2-cae0-46b6-b1a8-b33db60f8608\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:45.901989 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.901958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5l5b\" (UniqueName: \"kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b\") pod \"limitador-operator-controller-manager-74495c69f8-g8l6h\" (UID: \"c773d4a2-cae0-46b6-b1a8-b33db60f8608\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:45.920986 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:45.920963    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5l5b\" (UniqueName: \"kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b\") pod \"limitador-operator-controller-manager-74495c69f8-g8l6h\" (UID: \"c773d4a2-cae0-46b6-b1a8-b33db60f8608\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:46.047901 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:46.047831    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:46.326126 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:46.326091    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"]
Sep 12 01:52:46.326980 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:52:46.326956    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc773d4a2_cae0_46b6_b1a8_b33db60f8608.slice/crio-08965c96f8a7c1b24f6a9f44baf4788b85674fe2d207329e651e8596f5b7e632 WatchSource:0}: Error finding container 08965c96f8a7c1b24f6a9f44baf4788b85674fe2d207329e651e8596f5b7e632: Status 404 returned error can't find the container with id 08965c96f8a7c1b24f6a9f44baf4788b85674fe2d207329e651e8596f5b7e632
Sep 12 01:52:46.372063 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:46.372021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h" event={"ID":"c773d4a2-cae0-46b6-b1a8-b33db60f8608","Type":"ContainerStarted","Data":"08965c96f8a7c1b24f6a9f44baf4788b85674fe2d207329e651e8596f5b7e632"}
Sep 12 01:52:49.386783 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:49.386747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h" event={"ID":"c773d4a2-cae0-46b6-b1a8-b33db60f8608","Type":"ContainerStarted","Data":"75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c"}
Sep 12 01:52:49.387173 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:49.386868    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:52:49.411311 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:52:49.411269    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h" podStartSLOduration=1.751877095 podStartE2EDuration="4.411256393s" totalImagesPullingTime="2.659379298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:52:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:52:49.409687374 +0000 UTC m=+463.336633259" watchObservedRunningTime="2026-09-12 01:52:49.411256393 +0000 UTC m=+463.338202279"
Sep 12 01:53:00.393507 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:00.393467    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:11.018647 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.018611    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"]
Sep 12 01:53:11.021717 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.021700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.025026 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.025007    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-snrcf\""
Sep 12 01:53:11.039104 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.039075    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"]
Sep 12 01:53:11.188876 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.188836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.189087 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.188887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqjq\" (UniqueName: \"kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.290159 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.290085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqjq\" (UniqueName: \"kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.290159 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.290147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.290487 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.290469    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.299226 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.299204    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqjq\" (UniqueName: \"kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq\") pod \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.336118 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.336089    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:11.613761 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.613736    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"]
Sep 12 01:53:11.615968 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:53:11.615943    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84eda114_b8fd_4469_9dae_a1d3e759324b.slice/crio-bee3606cb491f3e313824883c3d4382ac34772921926d7e19e09588024f6ae53 WatchSource:0}: Error finding container bee3606cb491f3e313824883c3d4382ac34772921926d7e19e09588024f6ae53: Status 404 returned error can't find the container with id bee3606cb491f3e313824883c3d4382ac34772921926d7e19e09588024f6ae53
Sep 12 01:53:11.715170 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.715136    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"]
Sep 12 01:53:11.719865 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.719839    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"]
Sep 12 01:53:11.752980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.752954    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:11.757698 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.757664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:11.759050 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.759015    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"]
Sep 12 01:53:11.759240 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.759213    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" containerName="manager" containerID="cri-o://75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c" gracePeriod=2
Sep 12 01:53:11.780843 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.780809    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:11.782761 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.782724    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"]
Sep 12 01:53:11.814747 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:11.814717    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g8l6h\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:11.894400 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.894373    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlvnc\" (UniqueName: \"kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:11.894521 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.894428    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:11.978279 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.978256    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:11.980494 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:11.980471    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g8l6h\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:11.995815 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.995790    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlvnc\" (UniqueName: \"kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:11.995929 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.995836    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:11.996175 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:11.996158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:12.006524 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.006504    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlvnc\" (UniqueName: \"kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc\") pod \"kuadrant-operator-controller-manager-5f685c496f-fmjts\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:12.096189 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.096166    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5l5b\" (UniqueName: \"kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b\") pod \"c773d4a2-cae0-46b6-b1a8-b33db60f8608\" (UID: \"c773d4a2-cae0-46b6-b1a8-b33db60f8608\") "
Sep 12 01:53:12.098081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.098063    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b" (OuterVolumeSpecName: "kube-api-access-s5l5b") pod "c773d4a2-cae0-46b6-b1a8-b33db60f8608" (UID: "c773d4a2-cae0-46b6-b1a8-b33db60f8608"). InnerVolumeSpecName "kube-api-access-s5l5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:53:12.119922 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.119904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:12.196685 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.196655    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5l5b\" (UniqueName: \"kubernetes.io/projected/c773d4a2-cae0-46b6-b1a8-b33db60f8608-kube-api-access-s5l5b\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:53:12.436326 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.436245    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:12.441637 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:53:12.441606    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb150172_23b2_4fc8_82b4_b8199e35e44a.slice/crio-18acfb9319e096375b8be5db174db7bb96f329a7554f8a115ba7bb60c06fdf62 WatchSource:0}: Error finding container 18acfb9319e096375b8be5db174db7bb96f329a7554f8a115ba7bb60c06fdf62: Status 404 returned error can't find the container with id 18acfb9319e096375b8be5db174db7bb96f329a7554f8a115ba7bb60c06fdf62
Sep 12 01:53:12.480556 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.480524    2582 generic.go:356] "Generic (PLEG): container finished" podID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" containerID="75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c" exitCode=0
Sep 12 01:53:12.480678 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.480581    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:12.480678 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.480633    2582 scope.go:117] "RemoveContainer" containerID="75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c"
Sep 12 01:53:12.483355 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:12.483324    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g8l6h\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:12.484317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.484291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" event={"ID":"cb150172-23b2-4fc8-82b4-b8199e35e44a","Type":"ContainerStarted","Data":"18acfb9319e096375b8be5db174db7bb96f329a7554f8a115ba7bb60c06fdf62"}
Sep 12 01:53:12.490144 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.490125    2582 scope.go:117] "RemoveContainer" containerID="75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c"
Sep 12 01:53:12.490451 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:12.490424    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c\": container with ID starting with 75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c not found: ID does not exist" containerID="75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c"
Sep 12 01:53:12.490526 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.490460    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c"} err="failed to get container status \"75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c\": rpc error: code = NotFound desc = could not find container \"75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c\": container with ID starting with 75194407d12500a0fcc18f73ef7d0dd96512db9f4c828a6a1bc87accfb6d132c not found: ID does not exist"
Sep 12 01:53:12.494360 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:12.494334    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-g8l6h\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-g8l6h"
Sep 12 01:53:12.555740 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:12.555719    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" path="/var/lib/kubelet/pods/c773d4a2-cae0-46b6-b1a8-b33db60f8608/volumes"
Sep 12 01:53:16.506686 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.506623    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" containerName="manager" containerID="cri-o://d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539" gracePeriod=2
Sep 12 01:53:16.508228 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.508199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" event={"ID":"cb150172-23b2-4fc8-82b4-b8199e35e44a","Type":"ContainerStarted","Data":"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"}
Sep 12 01:53:16.508339 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.508302    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:16.531194 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.531154    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" podStartSLOduration=2.426058136 podStartE2EDuration="5.531140661s" totalImagesPullingTime="3.105082525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:16.528869835 +0000 UTC m=+490.455815731" watchObservedRunningTime="2026-09-12 01:53:16.531140661 +0000 UTC m=+490.458086546"
Sep 12 01:53:16.731736 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.731708    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:16.835533 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.835447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume\") pod \"84eda114-b8fd-4469-9dae-a1d3e759324b\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") "
Sep 12 01:53:16.835704 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.835573    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtqjq\" (UniqueName: \"kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq\") pod \"84eda114-b8fd-4469-9dae-a1d3e759324b\" (UID: \"84eda114-b8fd-4469-9dae-a1d3e759324b\") "
Sep 12 01:53:16.835780 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.835751    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "84eda114-b8fd-4469-9dae-a1d3e759324b" (UID: "84eda114-b8fd-4469-9dae-a1d3e759324b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:53:16.837600 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.837570    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq" (OuterVolumeSpecName: "kube-api-access-dtqjq") pod "84eda114-b8fd-4469-9dae-a1d3e759324b" (UID: "84eda114-b8fd-4469-9dae-a1d3e759324b"). InnerVolumeSpecName "kube-api-access-dtqjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:53:16.936937 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.936903    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtqjq\" (UniqueName: \"kubernetes.io/projected/84eda114-b8fd-4469-9dae-a1d3e759324b-kube-api-access-dtqjq\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:53:16.936937 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:16.936930    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/84eda114-b8fd-4469-9dae-a1d3e759324b-extensions-socket-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:53:17.514732 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:17.514697    2582 generic.go:356] "Generic (PLEG): container finished" podID="84eda114-b8fd-4469-9dae-a1d3e759324b" containerID="d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539" exitCode=2
Sep 12 01:53:17.515165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:17.514741    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:17.515165 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:17.514799    2582 scope.go:117] "RemoveContainer" containerID="d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539"
Sep 12 01:53:17.517172 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:17.517148    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:17.523742 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:17.523727    2582 scope.go:117] "RemoveContainer" containerID="d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539"
Sep 12 01:53:17.523992 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:17.523972    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539\": container with ID starting with d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539 not found: ID does not exist" containerID="d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539"
Sep 12 01:53:17.524054 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:17.523999    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539"} err="failed to get container status \"d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539\": rpc error: code = NotFound desc = could not find container \"d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539\": container with ID starting with d247b3ca972166787bcf3b5b3e565c1ea33f556a7e90cd94ab1f002aa4f25539 not found: ID does not exist"
Sep 12 01:53:17.525683 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:17.525661    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-xxks9\" is forbidden: User \"system:node:ip-10-0-128-81.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-81.ec2.internal' and this object" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-xxks9"
Sep 12 01:53:18.553523 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:18.553489    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" path="/var/lib/kubelet/pods/84eda114-b8fd-4469-9dae-a1d3e759324b/volumes"
Sep 12 01:53:27.517938 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:27.517905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:39.650142 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:39.650105    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:39.650516 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:39.650310    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" podUID="cb150172-23b2-4fc8-82b4-b8199e35e44a" containerName="manager" containerID="cri-o://2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5" gracePeriod=10
Sep 12 01:53:39.891443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:39.891421    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:40.001126 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.001030    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume\") pod \"cb150172-23b2-4fc8-82b4-b8199e35e44a\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") "
Sep 12 01:53:40.001126 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.001103    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlvnc\" (UniqueName: \"kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc\") pod \"cb150172-23b2-4fc8-82b4-b8199e35e44a\" (UID: \"cb150172-23b2-4fc8-82b4-b8199e35e44a\") "
Sep 12 01:53:40.001458 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.001436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cb150172-23b2-4fc8-82b4-b8199e35e44a" (UID: "cb150172-23b2-4fc8-82b4-b8199e35e44a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 01:53:40.003220 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.003195    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc" (OuterVolumeSpecName: "kube-api-access-hlvnc") pod "cb150172-23b2-4fc8-82b4-b8199e35e44a" (UID: "cb150172-23b2-4fc8-82b4-b8199e35e44a"). InnerVolumeSpecName "kube-api-access-hlvnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:53:40.102074 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.102051    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlvnc\" (UniqueName: \"kubernetes.io/projected/cb150172-23b2-4fc8-82b4-b8199e35e44a-kube-api-access-hlvnc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:53:40.102074 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.102072    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cb150172-23b2-4fc8-82b4-b8199e35e44a-extensions-socket-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:53:40.607697 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.607666    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb150172-23b2-4fc8-82b4-b8199e35e44a" containerID="2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5" exitCode=0
Sep 12 01:53:40.607833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.607710    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"
Sep 12 01:53:40.607833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.607717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" event={"ID":"cb150172-23b2-4fc8-82b4-b8199e35e44a","Type":"ContainerDied","Data":"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"}
Sep 12 01:53:40.607833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.607759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts" event={"ID":"cb150172-23b2-4fc8-82b4-b8199e35e44a","Type":"ContainerDied","Data":"18acfb9319e096375b8be5db174db7bb96f329a7554f8a115ba7bb60c06fdf62"}
Sep 12 01:53:40.607833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.607781    2582 scope.go:117] "RemoveContainer" containerID="2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"
Sep 12 01:53:40.616067 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.616050    2582 scope.go:117] "RemoveContainer" containerID="2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"
Sep 12 01:53:40.616338 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:53:40.616318    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5\": container with ID starting with 2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5 not found: ID does not exist" containerID="2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"
Sep 12 01:53:40.616415 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.616345    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5"} err="failed to get container status \"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5\": rpc error: code = NotFound desc = could not find container \"2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5\": container with ID starting with 2cb10bd752aef9c0874f69e4375920855928d7fa4ba50699860ce2e49697bca5 not found: ID does not exist"
Sep 12 01:53:40.628810 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.628785    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:40.634943 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:40.634919    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-fmjts"]
Sep 12 01:53:42.554656 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:42.554627    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb150172-23b2-4fc8-82b4-b8199e35e44a" path="/var/lib/kubelet/pods/cb150172-23b2-4fc8-82b4-b8199e35e44a/volumes"
Sep 12 01:53:44.010315 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010281    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"]
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" containerName="manager"
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010575    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" containerName="manager"
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010586    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb150172-23b2-4fc8-82b4-b8199e35e44a" containerName="manager"
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010595    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb150172-23b2-4fc8-82b4-b8199e35e44a" containerName="manager"
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010610    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" containerName="manager"
Sep 12 01:53:44.010661 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010619    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" containerName="manager"
Sep 12 01:53:44.010840 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010694    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb150172-23b2-4fc8-82b4-b8199e35e44a" containerName="manager"
Sep 12 01:53:44.010840 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010710    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c773d4a2-cae0-46b6-b1a8-b33db60f8608" containerName="manager"
Sep 12 01:53:44.010840 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.010720    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="84eda114-b8fd-4469-9dae-a1d3e759324b" containerName="manager"
Sep 12 01:53:44.015106 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.015084    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.017546 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.017523    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-84thm\""
Sep 12 01:53:44.025450 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.025428    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"]
Sep 12 01:53:44.129692 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129692 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjlj\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-kube-api-access-tzjlj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129851    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.129944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.129865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230455 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjlj\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-kube-api-access-tzjlj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230634    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230897 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230659    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230897 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.230897 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230846    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.231071 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.230983    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.231113 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.231078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.231523 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.231493    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.231664 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.231470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.232211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.232188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.236944 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.236916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.239534 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.239452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.240138 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.240087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.240759 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.240740    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjlj\" (UniqueName: \"kubernetes.io/projected/fea56fa2-4cc6-4ed1-b839-a8ccad4825e3-kube-api-access-tzjlj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-qk9nh\" (UID: \"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.326762 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.326685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:44.597182 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.597102    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"]
Sep 12 01:53:44.599849 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:53:44.599819    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea56fa2_4cc6_4ed1_b839_a8ccad4825e3.slice/crio-5ef839b99b14edb482cb473b3955c4d6f44714c9beb7454d10ecb0f1bca6622d WatchSource:0}: Error finding container 5ef839b99b14edb482cb473b3955c4d6f44714c9beb7454d10ecb0f1bca6622d: Status 404 returned error can't find the container with id 5ef839b99b14edb482cb473b3955c4d6f44714c9beb7454d10ecb0f1bca6622d
Sep 12 01:53:44.626228 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:44.626200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" event={"ID":"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3","Type":"ContainerStarted","Data":"5ef839b99b14edb482cb473b3955c4d6f44714c9beb7454d10ecb0f1bca6622d"}
Sep 12 01:53:47.285657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:47.285619    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:53:47.285938 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:47.285703    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:53:47.285938 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:47.285731    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:53:47.643662 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:47.643627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" event={"ID":"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3","Type":"ContainerStarted","Data":"ae63e979e8299d66ca3db09b8a244479e472536c9bda65a96b8ef086ca1d5960"}
Sep 12 01:53:47.664982 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:47.664936    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podStartSLOduration=1.98137081 podStartE2EDuration="4.664920347s" totalImagesPullingTime="2.683549537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:47.663902572 +0000 UTC m=+521.590848470" watchObservedRunningTime="2026-09-12 01:53:47.664920347 +0000 UTC m=+521.591866297"
Sep 12 01:53:48.327845 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:48.327812    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:48.332322 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:48.332300    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:48.647690 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:48.647658    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:48.648668 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:48.648646    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:53:56.009559 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.009518    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:53:56.011765 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.011746    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.015688 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.015663    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bf5qb\""
Sep 12 01:53:56.015837 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.015707    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 12 01:53:56.016250 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.016231    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 12 01:53:56.016250 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.016244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 12 01:53:56.020814 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.020785    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:53:56.108010 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.107972    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:53:56.120448 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.120411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.120609 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.120499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdjc\" (UniqueName: \"kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.221778 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.221739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdjc\" (UniqueName: \"kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.221929 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.221841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.222517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.222487    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.229997 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.229964    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdjc\" (UniqueName: \"kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc\") pod \"limitador-limitador-8588749977-sh7qt\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") " pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.322533 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.322473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:56.588196 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.588172    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:53:56.590790 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:53:56.590760    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod635fad27_0e18_48ef_b0cb_0628c37228e6.slice/crio-166a25d54fcbb6b75b260e76a7d9b65abf3a31a511bd79a851915af1d9419dd9 WatchSource:0}: Error finding container 166a25d54fcbb6b75b260e76a7d9b65abf3a31a511bd79a851915af1d9419dd9: Status 404 returned error can't find the container with id 166a25d54fcbb6b75b260e76a7d9b65abf3a31a511bd79a851915af1d9419dd9
Sep 12 01:53:56.681261 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.681228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" event={"ID":"635fad27-0e18-48ef-b0cb-0628c37228e6","Type":"ContainerStarted","Data":"166a25d54fcbb6b75b260e76a7d9b65abf3a31a511bd79a851915af1d9419dd9"}
Sep 12 01:53:56.964116 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.964087    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:53:56.966794 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.966769    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:53:56.969108 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.969084    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-56m6m\""
Sep 12 01:53:56.974553 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:56.974530    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:53:57.027022 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.026993    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2fjw\" (UniqueName: \"kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw\") pod \"authorino-59c974bd48-2csm8\" (UID: \"614bc27a-2bba-4b85-bdbb-7e5580e1e77a\") " pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:53:57.128200 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.128165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2fjw\" (UniqueName: \"kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw\") pod \"authorino-59c974bd48-2csm8\" (UID: \"614bc27a-2bba-4b85-bdbb-7e5580e1e77a\") " pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:53:57.137368 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.137342    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2fjw\" (UniqueName: \"kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw\") pod \"authorino-59c974bd48-2csm8\" (UID: \"614bc27a-2bba-4b85-bdbb-7e5580e1e77a\") " pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:53:57.276299 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.276224    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:53:57.544114 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.544089    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:53:57.548584 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:53:57.548550    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod614bc27a_2bba_4b85_bdbb_7e5580e1e77a.slice/crio-45beeae4ce64e4d0e5c428f23da2e381cc052a12cd9f5564c8e177dca504eee4 WatchSource:0}: Error finding container 45beeae4ce64e4d0e5c428f23da2e381cc052a12cd9f5564c8e177dca504eee4: Status 404 returned error can't find the container with id 45beeae4ce64e4d0e5c428f23da2e381cc052a12cd9f5564c8e177dca504eee4
Sep 12 01:53:57.686552 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.686519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2csm8" event={"ID":"614bc27a-2bba-4b85-bdbb-7e5580e1e77a","Type":"ContainerStarted","Data":"45beeae4ce64e4d0e5c428f23da2e381cc052a12cd9f5564c8e177dca504eee4"}
Sep 12 01:53:57.687997 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.687970    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" event={"ID":"635fad27-0e18-48ef-b0cb-0628c37228e6","Type":"ContainerStarted","Data":"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"}
Sep 12 01:53:57.688137 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.688098    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:53:57.707323 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:57.707278    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" podStartSLOduration=1.747816069 podStartE2EDuration="2.707264429s" totalImagesPullingTime="959.44836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:57.706489373 +0000 UTC m=+531.633435261" watchObservedRunningTime="2026-09-12 01:53:57.707264429 +0000 UTC m=+531.634210349"
Sep 12 01:53:59.698818 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:59.698773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2csm8" event={"ID":"614bc27a-2bba-4b85-bdbb-7e5580e1e77a","Type":"ContainerStarted","Data":"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"}
Sep 12 01:53:59.715373 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:53:59.715326    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-2csm8" podStartSLOduration=2.206805204 podStartE2EDuration="3.715310977s" totalImagesPullingTime="1.508505773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:53:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:53:59.713849302 +0000 UTC m=+533.640795188" watchObservedRunningTime="2026-09-12 01:53:59.715310977 +0000 UTC m=+533.642256863"
Sep 12 01:54:08.694001 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:08.693966    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:54:09.376517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:09.376483    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:54:09.376704 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:09.376684    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" podUID="635fad27-0e18-48ef-b0cb-0628c37228e6" containerName="limitador" containerID="cri-o://73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff" gracePeriod=30
Sep 12 01:54:09.919652 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:09.919626    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:54:10.020877 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.020814    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpdjc\" (UniqueName: \"kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc\") pod \"635fad27-0e18-48ef-b0cb-0628c37228e6\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") "
Sep 12 01:54:10.020877 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.020872    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file\") pod \"635fad27-0e18-48ef-b0cb-0628c37228e6\" (UID: \"635fad27-0e18-48ef-b0cb-0628c37228e6\") "
Sep 12 01:54:10.021243 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.021219    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file" (OuterVolumeSpecName: "config-file") pod "635fad27-0e18-48ef-b0cb-0628c37228e6" (UID: "635fad27-0e18-48ef-b0cb-0628c37228e6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:54:10.022751 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.022725    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc" (OuterVolumeSpecName: "kube-api-access-kpdjc") pod "635fad27-0e18-48ef-b0cb-0628c37228e6" (UID: "635fad27-0e18-48ef-b0cb-0628c37228e6"). InnerVolumeSpecName "kube-api-access-kpdjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:54:10.122255 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.122226    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpdjc\" (UniqueName: \"kubernetes.io/projected/635fad27-0e18-48ef-b0cb-0628c37228e6-kube-api-access-kpdjc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:10.122255 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.122250    2582 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/635fad27-0e18-48ef-b0cb-0628c37228e6-config-file\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:10.743673 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.743642    2582 generic.go:356] "Generic (PLEG): container finished" podID="635fad27-0e18-48ef-b0cb-0628c37228e6" containerID="73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff" exitCode=0
Sep 12 01:54:10.743827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.743681    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt"
Sep 12 01:54:10.743827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.743708    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" event={"ID":"635fad27-0e18-48ef-b0cb-0628c37228e6","Type":"ContainerDied","Data":"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"}
Sep 12 01:54:10.743827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.743748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-sh7qt" event={"ID":"635fad27-0e18-48ef-b0cb-0628c37228e6","Type":"ContainerDied","Data":"166a25d54fcbb6b75b260e76a7d9b65abf3a31a511bd79a851915af1d9419dd9"}
Sep 12 01:54:10.743827 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.743769    2582 scope.go:117] "RemoveContainer" containerID="73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"
Sep 12 01:54:10.751635 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.751588    2582 scope.go:117] "RemoveContainer" containerID="73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"
Sep 12 01:54:10.751882 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:54:10.751863    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff\": container with ID starting with 73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff not found: ID does not exist" containerID="73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"
Sep 12 01:54:10.751940 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.751890    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff"} err="failed to get container status \"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff\": rpc error: code = NotFound desc = could not find container \"73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff\": container with ID starting with 73d33d4dd7f04b0d7ed31545d6b52c07d55b1183c09443024a07c35c215815ff not found: ID does not exist"
Sep 12 01:54:10.760145 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.760124    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:54:10.762941 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:10.762919    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-sh7qt"]
Sep 12 01:54:12.554388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:12.554350    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="635fad27-0e18-48ef-b0cb-0628c37228e6" path="/var/lib/kubelet/pods/635fad27-0e18-48ef-b0cb-0628c37228e6/volumes"
Sep 12 01:54:15.589282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.589252    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"]
Sep 12 01:54:15.589654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.589524    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="635fad27-0e18-48ef-b0cb-0628c37228e6" containerName="limitador"
Sep 12 01:54:15.589654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.589535    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="635fad27-0e18-48ef-b0cb-0628c37228e6" containerName="limitador"
Sep 12 01:54:15.589654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.589583    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="635fad27-0e18-48ef-b0cb-0628c37228e6" containerName="limitador"
Sep 12 01:54:15.593828 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.593809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.596439 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.596410    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 12 01:54:15.596581 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.596505    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 12 01:54:15.596644 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.596617    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 12 01:54:15.596694 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.596664    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-vf69j\""
Sep 12 01:54:15.602924 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.602901    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"]
Sep 12 01:54:15.660363 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.660336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb0334e8-2748-426f-bf56-86867b14619a-data\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.660491 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.660402    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6v2\" (UniqueName: \"kubernetes.io/projected/fb0334e8-2748-426f-bf56-86867b14619a-kube-api-access-5m6v2\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.761234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.761207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb0334e8-2748-426f-bf56-86867b14619a-data\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.761373 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.761270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6v2\" (UniqueName: \"kubernetes.io/projected/fb0334e8-2748-426f-bf56-86867b14619a-kube-api-access-5m6v2\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.761634 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.761616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fb0334e8-2748-426f-bf56-86867b14619a-data\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.769660 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.769642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6v2\" (UniqueName: \"kubernetes.io/projected/fb0334e8-2748-426f-bf56-86867b14619a-kube-api-access-5m6v2\") pod \"postgres-69fb4594cf-bk7j7\" (UID: \"fb0334e8-2748-426f-bf56-86867b14619a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:15.905202 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:15.905165    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:16.175128 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:16.175105    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"]
Sep 12 01:54:16.177218 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:54:16.177189    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb0334e8_2748_426f_bf56_86867b14619a.slice/crio-08677ea847d3bbe09cee662a5dc869afb5370bdfaaa3ef7583e82532598999e5 WatchSource:0}: Error finding container 08677ea847d3bbe09cee662a5dc869afb5370bdfaaa3ef7583e82532598999e5: Status 404 returned error can't find the container with id 08677ea847d3bbe09cee662a5dc869afb5370bdfaaa3ef7583e82532598999e5
Sep 12 01:54:16.768867 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:16.768815    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7" event={"ID":"fb0334e8-2748-426f-bf56-86867b14619a","Type":"ContainerStarted","Data":"08677ea847d3bbe09cee662a5dc869afb5370bdfaaa3ef7583e82532598999e5"}
Sep 12 01:54:21.792112 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:21.792074    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7" event={"ID":"fb0334e8-2748-426f-bf56-86867b14619a","Type":"ContainerStarted","Data":"6d8f605c944fa801bca1e9d436d86f61de0eadc7658476688f953800051bcb55"}
Sep 12 01:54:21.792562 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:21.792185    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:21.808676 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:21.808617    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7" podStartSLOduration=2.129267143 podStartE2EDuration="6.808598129s" totalImagesPullingTime="4.679330986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:54:21.807887624 +0000 UTC m=+555.734833716" watchObservedRunningTime="2026-09-12 01:54:21.808598129 +0000 UTC m=+555.735544015"
Sep 12 01:54:27.824772 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:27.824740    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-bk7j7"
Sep 12 01:54:28.682654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.682621    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bd64c89c6-jvlwb"]
Sep 12 01:54:28.685741 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.685722    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.694417 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.694395    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bd64c89c6-jvlwb"]
Sep 12 01:54:28.728627 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.728601    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd64c89c6-jvlwb"]
Sep 12 01:54:28.728785 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:54:28.728767    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vxwzx], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7bd64c89c6-jvlwb" podUID="0dce327b-5c4d-4f73-ad06-e73595ae68b9"
Sep 12 01:54:28.775179 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.775154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxwzx\" (UniqueName: \"kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx\") pod \"authorino-7bd64c89c6-jvlwb\" (UID: \"0dce327b-5c4d-4f73-ad06-e73595ae68b9\") " pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.819643 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.819620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.825862 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.825845    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.875460 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.875434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxwzx\" (UniqueName: \"kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx\") pod \"authorino-7bd64c89c6-jvlwb\" (UID: \"0dce327b-5c4d-4f73-ad06-e73595ae68b9\") " pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.883387 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.883362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxwzx\" (UniqueName: \"kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx\") pod \"authorino-7bd64c89c6-jvlwb\" (UID: \"0dce327b-5c4d-4f73-ad06-e73595ae68b9\") " pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:28.975757 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.975706    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxwzx\" (UniqueName: \"kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx\") pod \"0dce327b-5c4d-4f73-ad06-e73595ae68b9\" (UID: \"0dce327b-5c4d-4f73-ad06-e73595ae68b9\") "
Sep 12 01:54:28.977560 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:28.977529    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx" (OuterVolumeSpecName: "kube-api-access-vxwzx") pod "0dce327b-5c4d-4f73-ad06-e73595ae68b9" (UID: "0dce327b-5c4d-4f73-ad06-e73595ae68b9"). InnerVolumeSpecName "kube-api-access-vxwzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:54:29.076733 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:29.076711    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxwzx\" (UniqueName: \"kubernetes.io/projected/0dce327b-5c4d-4f73-ad06-e73595ae68b9-kube-api-access-vxwzx\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:29.824363 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:29.824334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bd64c89c6-jvlwb"
Sep 12 01:54:29.853383 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:29.853358    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bd64c89c6-jvlwb"]
Sep 12 01:54:29.856968 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:29.856946    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bd64c89c6-jvlwb"]
Sep 12 01:54:30.528638 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.528608    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:54:30.528796 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.528778    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-2csm8" podUID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" containerName="authorino" containerID="cri-o://52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27" gracePeriod=30
Sep 12 01:54:30.554798 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.554775    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0dce327b-5c4d-4f73-ad06-e73595ae68b9" path="/var/lib/kubelet/pods/0dce327b-5c4d-4f73-ad06-e73595ae68b9/volumes"
Sep 12 01:54:30.763521 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.763494    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:54:30.829215 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.829136    2582 generic.go:356] "Generic (PLEG): container finished" podID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" containerID="52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27" exitCode=0
Sep 12 01:54:30.829215 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.829178    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-2csm8"
Sep 12 01:54:30.829393 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.829216    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2csm8" event={"ID":"614bc27a-2bba-4b85-bdbb-7e5580e1e77a","Type":"ContainerDied","Data":"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"}
Sep 12 01:54:30.829393 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.829256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-2csm8" event={"ID":"614bc27a-2bba-4b85-bdbb-7e5580e1e77a","Type":"ContainerDied","Data":"45beeae4ce64e4d0e5c428f23da2e381cc052a12cd9f5564c8e177dca504eee4"}
Sep 12 01:54:30.829393 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.829274    2582 scope.go:117] "RemoveContainer" containerID="52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"
Sep 12 01:54:30.836694 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.836678    2582 scope.go:117] "RemoveContainer" containerID="52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"
Sep 12 01:54:30.836953 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:54:30.836934    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27\": container with ID starting with 52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27 not found: ID does not exist" containerID="52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"
Sep 12 01:54:30.837023 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.836966    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27"} err="failed to get container status \"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27\": rpc error: code = NotFound desc = could not find container \"52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27\": container with ID starting with 52aa9fd28dfa0416c200680774740a2b8b4fed4fc6f55714e6e22f8b9bb3bf27 not found: ID does not exist"
Sep 12 01:54:30.892763 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.892737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2fjw\" (UniqueName: \"kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw\") pod \"614bc27a-2bba-4b85-bdbb-7e5580e1e77a\" (UID: \"614bc27a-2bba-4b85-bdbb-7e5580e1e77a\") "
Sep 12 01:54:30.894695 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.894672    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw" (OuterVolumeSpecName: "kube-api-access-v2fjw") pod "614bc27a-2bba-4b85-bdbb-7e5580e1e77a" (UID: "614bc27a-2bba-4b85-bdbb-7e5580e1e77a"). InnerVolumeSpecName "kube-api-access-v2fjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:54:30.993844 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:30.993821    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2fjw\" (UniqueName: \"kubernetes.io/projected/614bc27a-2bba-4b85-bdbb-7e5580e1e77a-kube-api-access-v2fjw\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:31.156441 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:31.156404    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:54:31.159392 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:31.159365    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-2csm8"]
Sep 12 01:54:32.553826 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:32.553793    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" path="/var/lib/kubelet/pods/614bc27a-2bba-4b85-bdbb-7e5580e1e77a/volumes"
Sep 12 01:54:35.597805 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.597776    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:35.598276 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.598217    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" containerName="authorino"
Sep 12 01:54:35.598276 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.598237    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" containerName="authorino"
Sep 12 01:54:35.598388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.598315    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="614bc27a-2bba-4b85-bdbb-7e5580e1e77a" containerName="authorino"
Sep 12 01:54:35.603894 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.603871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.606795 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.606775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 01:54:35.607109 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.607094    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 01:54:35.607838 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.607821    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 01:54:35.607910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.607849    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d79dj\""
Sep 12 01:54:35.621544 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.621519    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:35.727820 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.727788    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.728060 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.727917    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh92s\" (UniqueName: \"kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.728060 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.727956    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.814671 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.814644    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 01:54:35.818584 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.818566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:35.828345 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.828320    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh92s\" (UniqueName: \"kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.828451 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.828352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.828451 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.828357    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 01:54:35.828451 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.828389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.830995 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.830973    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.831099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.830973    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.836675 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.836651    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh92s\" (UniqueName: \"kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s\") pod \"maas-controller-75c5588db8-slp5x\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") " pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.914538 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.914504    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:35.929451 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.929427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:35.929549 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.929465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4sxd\" (UniqueName: \"kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:35.929549 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:35.929506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.030120 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.030089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.030263 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.030152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.030263 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.030188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4sxd\" (UniqueName: \"kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.032591 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.032570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.032657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.032570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.038133 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.038104    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4sxd\" (UniqueName: \"kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd\") pod \"maas-controller-6f875f6cb9-kpwgq\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") " pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.128946 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.128920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:36.183682 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.183140    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:36.188855 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:54:36.188823    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f8a6fc_712f_4b97_9fb7_a67336a32644.slice/crio-c77e5f4f5e1b2910840df6f30b6d8678a5990133d8f3c5eff8c27f3d24b991a7 WatchSource:0}: Error finding container c77e5f4f5e1b2910840df6f30b6d8678a5990133d8f3c5eff8c27f3d24b991a7: Status 404 returned error can't find the container with id c77e5f4f5e1b2910840df6f30b6d8678a5990133d8f3c5eff8c27f3d24b991a7
Sep 12 01:54:36.397516 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.397491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 01:54:36.399526 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:54:36.399498    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01126ec2_baf9_4e0f_8f60_3cb1a6b8f7e2.slice/crio-94b6fef22a1361aff6b1d054bfcd0b51ff38e5447e2c50f17b73ca2598667beb WatchSource:0}: Error finding container 94b6fef22a1361aff6b1d054bfcd0b51ff38e5447e2c50f17b73ca2598667beb: Status 404 returned error can't find the container with id 94b6fef22a1361aff6b1d054bfcd0b51ff38e5447e2c50f17b73ca2598667beb
Sep 12 01:54:36.855834 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.855801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5588db8-slp5x" event={"ID":"05f8a6fc-712f-4b97-9fb7-a67336a32644","Type":"ContainerStarted","Data":"c77e5f4f5e1b2910840df6f30b6d8678a5990133d8f3c5eff8c27f3d24b991a7"}
Sep 12 01:54:36.857460 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:36.857433    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" event={"ID":"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2","Type":"ContainerStarted","Data":"94b6fef22a1361aff6b1d054bfcd0b51ff38e5447e2c50f17b73ca2598667beb"}
Sep 12 01:54:39.873307 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.873269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" event={"ID":"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2","Type":"ContainerStarted","Data":"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"}
Sep 12 01:54:39.873705 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.873499    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:39.874842 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.874814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5588db8-slp5x" event={"ID":"05f8a6fc-712f-4b97-9fb7-a67336a32644","Type":"ContainerStarted","Data":"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"}
Sep 12 01:54:39.874980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.874948    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:39.892307 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.892269    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" podStartSLOduration=2.473125509 podStartE2EDuration="4.892252154s" totalImagesPullingTime="2.419126645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:54:39.89074681 +0000 UTC m=+573.817692695" watchObservedRunningTime="2026-09-12 01:54:39.892252154 +0000 UTC m=+573.819198041"
Sep 12 01:54:39.908645 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:39.908607    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-75c5588db8-slp5x" podStartSLOduration=2.2884283229999998 podStartE2EDuration="4.90859327s" totalImagesPullingTime="2.620164947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:54:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:54:39.908081706 +0000 UTC m=+573.835027594" watchObservedRunningTime="2026-09-12 01:54:39.90859327 +0000 UTC m=+573.835539156"
Sep 12 01:54:50.880298 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:50.880262    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 01:54:50.880714 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:50.880344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:50.945759 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:50.945720    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:50.945941 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:50.945909    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-75c5588db8-slp5x" podUID="05f8a6fc-712f-4b97-9fb7-a67336a32644" containerName="manager" containerID="cri-o://b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b" gracePeriod=30
Sep 12 01:54:51.181518 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.181492    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:51.248234 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.248208    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh92s\" (UniqueName: \"kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s\") pod \"05f8a6fc-712f-4b97-9fb7-a67336a32644\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") "
Sep 12 01:54:51.248381 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.248239    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert\") pod \"05f8a6fc-712f-4b97-9fb7-a67336a32644\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") "
Sep 12 01:54:51.248381 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.248271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs\") pod \"05f8a6fc-712f-4b97-9fb7-a67336a32644\" (UID: \"05f8a6fc-712f-4b97-9fb7-a67336a32644\") "
Sep 12 01:54:51.250423 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.250395    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "05f8a6fc-712f-4b97-9fb7-a67336a32644" (UID: "05f8a6fc-712f-4b97-9fb7-a67336a32644"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:54:51.250518 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.250428    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "05f8a6fc-712f-4b97-9fb7-a67336a32644" (UID: "05f8a6fc-712f-4b97-9fb7-a67336a32644"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 01:54:51.250518 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.250496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s" (OuterVolumeSpecName: "kube-api-access-lh92s") pod "05f8a6fc-712f-4b97-9fb7-a67336a32644" (UID: "05f8a6fc-712f-4b97-9fb7-a67336a32644"). InnerVolumeSpecName "kube-api-access-lh92s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:54:51.348663 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.348640    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:51.348663 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.348662    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh92s\" (UniqueName: \"kubernetes.io/projected/05f8a6fc-712f-4b97-9fb7-a67336a32644-kube-api-access-lh92s\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:51.348791 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.348672    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/05f8a6fc-712f-4b97-9fb7-a67336a32644-webhook-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:54:51.924682 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.924648    2582 generic.go:356] "Generic (PLEG): container finished" podID="05f8a6fc-712f-4b97-9fb7-a67336a32644" containerID="b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b" exitCode=0
Sep 12 01:54:51.925174 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.924703    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-75c5588db8-slp5x"
Sep 12 01:54:51.925174 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.924717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5588db8-slp5x" event={"ID":"05f8a6fc-712f-4b97-9fb7-a67336a32644","Type":"ContainerDied","Data":"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"}
Sep 12 01:54:51.925174 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.924758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-75c5588db8-slp5x" event={"ID":"05f8a6fc-712f-4b97-9fb7-a67336a32644","Type":"ContainerDied","Data":"c77e5f4f5e1b2910840df6f30b6d8678a5990133d8f3c5eff8c27f3d24b991a7"}
Sep 12 01:54:51.925174 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.924782    2582 scope.go:117] "RemoveContainer" containerID="b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"
Sep 12 01:54:51.933911 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.933892    2582 scope.go:117] "RemoveContainer" containerID="b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"
Sep 12 01:54:51.934200 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:54:51.934181    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b\": container with ID starting with b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b not found: ID does not exist" containerID="b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"
Sep 12 01:54:51.934272 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.934215    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b"} err="failed to get container status \"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b\": rpc error: code = NotFound desc = could not find container \"b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b\": container with ID starting with b6195410ff48d59d3e6bde0144ac813c2d6675126796ab391227d041caa88f7b not found: ID does not exist"
Sep 12 01:54:51.949087 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.949065    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:51.952949 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:51.952931    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-75c5588db8-slp5x"]
Sep 12 01:54:52.554018 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:54:52.553985    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05f8a6fc-712f-4b97-9fb7-a67336a32644" path="/var/lib/kubelet/pods/05f8a6fc-712f-4b97-9fb7-a67336a32644/volumes"
Sep 12 01:55:06.498279 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:06.498250    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:55:06.500580 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:06.500553    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 01:55:12.015929 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.015899    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 01:55:12.016323 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.016236    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05f8a6fc-712f-4b97-9fb7-a67336a32644" containerName="manager"
Sep 12 01:55:12.016323 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.016248    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="05f8a6fc-712f-4b97-9fb7-a67336a32644" containerName="manager"
Sep 12 01:55:12.016323 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.016305    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="05f8a6fc-712f-4b97-9fb7-a67336a32644" containerName="manager"
Sep 12 01:55:12.022315 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.022293    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.026081 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.024742    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 12 01:55:12.029992 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.029964    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 12 01:55:12.032068 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.032019    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 01:55:12.102805 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.102778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.102957 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.102814    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.102957 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.102831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.102957 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.102903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsqgt\" (UniqueName: \"kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.204292 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.204265    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.204443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.204302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.204443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.204318    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.204443 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.204425    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsqgt\" (UniqueName: \"kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.204609 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:55:12.204522    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found
Sep 12 01:55:12.204661 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:55:12.204618    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs podName:17d21534-a743-48e1-b3e2-269ed57d827c nodeName:}" failed. No retries permitted until 2026-09-12 01:55:12.704593343 +0000 UTC m=+606.631539222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs") pod "maas-api-77b9747d97-ztlmg" (UID: "17d21534-a743-48e1-b3e2-269ed57d827c") : secret "maas-api-metrics-tls" not found
Sep 12 01:55:12.204915 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.204895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.206812 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.206791    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.214909 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.214884    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsqgt\" (UniqueName: \"kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.619952 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.619917    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"]
Sep 12 01:55:12.623075 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.623059    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.625941 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.625916    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 12 01:55:12.643969 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.643943    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"]
Sep 12 01:55:12.664164 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.664138    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"]
Sep 12 01:55:12.667249 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.667235    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.670374 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.670354    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-j9jrv\""
Sep 12 01:55:12.687281 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.687261    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"]
Sep 12 01:55:12.709186 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.709166    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.709285 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.709209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.709326 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.709308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtv2\" (UniqueName: \"kubernetes.io/projected/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-kube-api-access-trtv2\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.709421 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.709397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc47836b-c450-47e4-a909-1bcdbec3b59e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.709510 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.709493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/cc47836b-c450-47e4-a909-1bcdbec3b59e-kube-api-access-tq6c7\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.711467 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.711444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") pod \"maas-api-77b9747d97-ztlmg\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") " pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.810483 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.810448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.810668 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.810496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtv2\" (UniqueName: \"kubernetes.io/projected/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-kube-api-access-trtv2\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.810668 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.810546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc47836b-c450-47e4-a909-1bcdbec3b59e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.810668 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.810615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/cc47836b-c450-47e4-a909-1bcdbec3b59e-kube-api-access-tq6c7\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.811288 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.811260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.811401 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.811380    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc47836b-c450-47e4-a909-1bcdbec3b59e-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.835134 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.835110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtv2\" (UniqueName: \"kubernetes.io/projected/00bf8136-c7c1-4807-a6f6-1ff68fccaff3-kube-api-access-trtv2\") pod \"payload-processing-5b54b7fbb7-nmhx2\" (UID: \"00bf8136-c7c1-4807-a6f6-1ff68fccaff3\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.835977 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.835955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq6c7\" (UniqueName: \"kubernetes.io/projected/cc47836b-c450-47e4-a909-1bcdbec3b59e-kube-api-access-tq6c7\") pod \"payload-pre-processing-59fbc47bf8-kwgmc\" (UID: \"cc47836b-c450-47e4-a909-1bcdbec3b59e\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:12.932688 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.932604    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:12.938413 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.938386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:12.976581 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:12.976402    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:13.210141 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:13.210113    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"]
Sep 12 01:55:13.212634 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:55:13.212600    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf8136_c7c1_4807_a6f6_1ff68fccaff3.slice/crio-fb40288ed3935a7369f131bc5770b3fc6728bc5a49ab7809a494ac8418f66782 WatchSource:0}: Error finding container fb40288ed3935a7369f131bc5770b3fc6728bc5a49ab7809a494ac8418f66782: Status 404 returned error can't find the container with id fb40288ed3935a7369f131bc5770b3fc6728bc5a49ab7809a494ac8418f66782
Sep 12 01:55:13.220164 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:13.220138    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 01:55:13.223643 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:55:13.223620    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d21534_a743_48e1_b3e2_269ed57d827c.slice/crio-a1c34ebbbc5222aea880869afe645e0dfe35e4810da48967641bfce62c29811d WatchSource:0}: Error finding container a1c34ebbbc5222aea880869afe645e0dfe35e4810da48967641bfce62c29811d: Status 404 returned error can't find the container with id a1c34ebbbc5222aea880869afe645e0dfe35e4810da48967641bfce62c29811d
Sep 12 01:55:13.281517 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:13.281491    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"]
Sep 12 01:55:13.283388 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:55:13.283361    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc47836b_c450_47e4_a909_1bcdbec3b59e.slice/crio-def96f99a223581627ada45f0d817f5337bc90f659b4d74cc341897df35450e9 WatchSource:0}: Error finding container def96f99a223581627ada45f0d817f5337bc90f659b4d74cc341897df35450e9: Status 404 returned error can't find the container with id def96f99a223581627ada45f0d817f5337bc90f659b4d74cc341897df35450e9
Sep 12 01:55:14.023262 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:14.023223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" event={"ID":"17d21534-a743-48e1-b3e2-269ed57d827c","Type":"ContainerStarted","Data":"a1c34ebbbc5222aea880869afe645e0dfe35e4810da48967641bfce62c29811d"}
Sep 12 01:55:14.025525 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:14.025447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc" event={"ID":"cc47836b-c450-47e4-a909-1bcdbec3b59e","Type":"ContainerStarted","Data":"def96f99a223581627ada45f0d817f5337bc90f659b4d74cc341897df35450e9"}
Sep 12 01:55:14.028485 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:14.028421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2" event={"ID":"00bf8136-c7c1-4807-a6f6-1ff68fccaff3","Type":"ContainerStarted","Data":"fb40288ed3935a7369f131bc5770b3fc6728bc5a49ab7809a494ac8418f66782"}
Sep 12 01:55:18.048282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.048243    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" event={"ID":"17d21534-a743-48e1-b3e2-269ed57d827c","Type":"ContainerStarted","Data":"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"}
Sep 12 01:55:18.048927 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.048909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:18.050077 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.050057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc" event={"ID":"cc47836b-c450-47e4-a909-1bcdbec3b59e","Type":"ContainerStarted","Data":"8b1f2b682b438192ba3907d47b3db829b40368f1c6100a342dd68b0a0bd97a4a"}
Sep 12 01:55:18.050416 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.050402    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:18.051724 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.051680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2" event={"ID":"00bf8136-c7c1-4807-a6f6-1ff68fccaff3","Type":"ContainerStarted","Data":"b2cefab5153e9f1306304e655d63a54f9ce9ebfaddf138f384de7e4cbedc4b9b"}
Sep 12 01:55:18.051829 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.051724    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:18.065971 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.065922    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" podStartSLOduration=3.282741169 podStartE2EDuration="7.065910192s" totalImagesPullingTime="3.783169023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:55:18.063883061 +0000 UTC m=+611.990828947" watchObservedRunningTime="2026-09-12 01:55:18.065910192 +0000 UTC m=+611.992856079"
Sep 12 01:55:18.081698 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.080258    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2" podStartSLOduration=2.225502185 podStartE2EDuration="6.080239452s" totalImagesPullingTime="3.854737267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:55:18.07909414 +0000 UTC m=+612.006040028" watchObservedRunningTime="2026-09-12 01:55:18.080239452 +0000 UTC m=+612.007185341"
Sep 12 01:55:18.098248 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:18.098193    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc" podStartSLOduration=2.320752769 podStartE2EDuration="6.098173882s" totalImagesPullingTime="3.777421113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:55:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:55:18.097547449 +0000 UTC m=+612.024493336" watchObservedRunningTime="2026-09-12 01:55:18.098173882 +0000 UTC m=+612.025119770"
Sep 12 01:55:22.073767 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.073681    2582 generic.go:356] "Generic (PLEG): container finished" podID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerID="ae63e979e8299d66ca3db09b8a244479e472536c9bda65a96b8ef086ca1d5960" exitCode=0
Sep 12 01:55:22.074212 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.073757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" event={"ID":"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3","Type":"ContainerDied","Data":"ae63e979e8299d66ca3db09b8a244479e472536c9bda65a96b8ef086ca1d5960"}
Sep 12 01:55:22.074212 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.074158    2582 scope.go:117] "RemoveContainer" containerID="ae63e979e8299d66ca3db09b8a244479e472536c9bda65a96b8ef086ca1d5960"
Sep 12 01:55:22.075228 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.075194    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:55:22.075330 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.075262    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:55:22.075330 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:22.075292    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:55:23.082119 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:23.082077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" event={"ID":"fea56fa2-4cc6-4ed1-b839-a8ccad4825e3","Type":"ContainerStarted","Data":"b240136d1b2478b5dc35f7ab4109df58d65e941c4ca12746c1d4cd6be0b6c2aa"}
Sep 12 01:55:23.327797 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:23.327761    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:55:23.328059 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:23.328016    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qk9nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 12 01:55:23.328113 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:23.328082    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podUID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 12 01:55:24.327053 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:24.327003    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qk9nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 12 01:55:24.327433 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:24.327073    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podUID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 12 01:55:25.067254 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:25.067223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 01:55:25.327105 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:25.327004    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qk9nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 12 01:55:25.327105 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:25.327093    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podUID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 12 01:55:26.327863 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:26.327825    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qk9nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 12 01:55:26.328265 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:26.327885    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podUID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 12 01:55:27.327655 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:27.327620    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-qk9nh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused" start-of-body=
Sep 12 01:55:27.327838 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:27.327673    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh" podUID="fea56fa2-4cc6-4ed1-b839-a8ccad4825e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.24:15021/healthz/ready\": dial tcp 10.134.0.24:15021: connect: connection refused"
Sep 12 01:55:28.330451 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:28.330417    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:55:28.330873 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:28.330686    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:55:28.331338 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:28.331321    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-qk9nh"
Sep 12 01:55:29.058910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.058838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-nmhx2"
Sep 12 01:55:29.625966 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.625937    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"]
Sep 12 01:55:29.628179 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.628156    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.631564 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.631541    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 12 01:55:29.631564 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.631559    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-dzfh7\""
Sep 12 01:55:29.631731 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.631620    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 12 01:55:29.631731 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.631548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 12 01:55:29.641008 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.640980    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"]
Sep 12 01:55:29.762350 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.762503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.762503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762429    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.762503 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vlt\" (UniqueName: \"kubernetes.io/projected/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-kube-api-access-58vlt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.762656 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.762656 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.762568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863288 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vlt\" (UniqueName: \"kubernetes.io/projected/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-kube-api-access-58vlt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863288 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863485 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863397    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863485 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863423    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863485 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863441    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863592 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863809 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863865 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.863915 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.863896    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.865794 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.865768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.865906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.865880    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.870684 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.870663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vlt\" (UniqueName: \"kubernetes.io/projected/c7fd7858-e180-4092-a21e-ee2e0e2d8e94-kube-api-access-58vlt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr\" (UID: \"c7fd7858-e180-4092-a21e-ee2e0e2d8e94\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:29.941567 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:29.941490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:30.064200 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:30.064169    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-kwgmc"
Sep 12 01:55:30.212408 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:30.212382    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"]
Sep 12 01:55:30.215002 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:55:30.214974    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fd7858_e180_4092_a21e_ee2e0e2d8e94.slice/crio-65e437612d11d20f328b4b949dafb9a2a7e2de42a804061e2c148511ec1660da WatchSource:0}: Error finding container 65e437612d11d20f328b4b949dafb9a2a7e2de42a804061e2c148511ec1660da: Status 404 returned error can't find the container with id 65e437612d11d20f328b4b949dafb9a2a7e2de42a804061e2c148511ec1660da
Sep 12 01:55:30.216339 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:30.216314    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 01:55:31.121744 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:31.121698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr" event={"ID":"c7fd7858-e180-4092-a21e-ee2e0e2d8e94","Type":"ContainerStarted","Data":"65e437612d11d20f328b4b949dafb9a2a7e2de42a804061e2c148511ec1660da"}
Sep 12 01:55:32.132580 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:32.132541    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr" event={"ID":"c7fd7858-e180-4092-a21e-ee2e0e2d8e94","Type":"ContainerStarted","Data":"4e44edd99259e8db0e069c09e3a9f6b51edf722700d289a8373757906706a709"}
Sep 12 01:55:32.152647 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:32.152604    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr" podStartSLOduration=2.063978229 podStartE2EDuration="3.152588441s" totalImagesPullingTime="1.088610212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:55:32.151464607 +0000 UTC m=+626.078410494" watchObservedRunningTime="2026-09-12 01:55:32.152588441 +0000 UTC m=+626.079534327"
Sep 12 01:55:39.941869 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:39.941833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:39.942321 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:39.941895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:39.954087 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:39.954061    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:40.178570 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:40.178544    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr"
Sep 12 01:55:48.822557 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.822521    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"]
Sep 12 01:55:48.827319 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.827298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.829629 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.829608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 12 01:55:48.836563 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.836540    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"]
Sep 12 01:55:48.921238 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5af3bbd-d620-4637-97a2-33e6acab602e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.921238 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.921447 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921267    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.921447 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921316    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.921447 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:48.921447 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:48.921427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmkkd\" (UniqueName: \"kubernetes.io/projected/f5af3bbd-d620-4637-97a2-33e6acab602e-kube-api-access-pmkkd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022260 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022267    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022414 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmkkd\" (UniqueName: \"kubernetes.io/projected/f5af3bbd-d620-4637-97a2-33e6acab602e-kube-api-access-pmkkd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022501 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5af3bbd-d620-4637-97a2-33e6acab602e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022501 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022601 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022651 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022609    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022718 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.022833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.022813    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.024675 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.024643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5af3bbd-d620-4637-97a2-33e6acab602e-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.024894 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.024877    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5af3bbd-d620-4637-97a2-33e6acab602e-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.030181 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.030157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmkkd\" (UniqueName: \"kubernetes.io/projected/f5af3bbd-d620-4637-97a2-33e6acab602e-kube-api-access-pmkkd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-d89mh\" (UID: \"f5af3bbd-d620-4637-97a2-33e6acab602e\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.137632 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.137583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:49.407349 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:49.407319    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"]
Sep 12 01:55:49.409616 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:55:49.409590    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5af3bbd_d620_4637_97a2_33e6acab602e.slice/crio-2463de37b2f82ffdea8aeede5cad7d22c54549e92c419bdb5810450eca93a353 WatchSource:0}: Error finding container 2463de37b2f82ffdea8aeede5cad7d22c54549e92c419bdb5810450eca93a353: Status 404 returned error can't find the container with id 2463de37b2f82ffdea8aeede5cad7d22c54549e92c419bdb5810450eca93a353
Sep 12 01:55:50.213553 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:50.213519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh" event={"ID":"f5af3bbd-d620-4637-97a2-33e6acab602e","Type":"ContainerStarted","Data":"c4a4d1cc7a9f001003fa58e72d251d9f4a3588264a21c0432d4f5ad0931a1974"}
Sep 12 01:55:50.213911 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:50.213572    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh" event={"ID":"f5af3bbd-d620-4637-97a2-33e6acab602e","Type":"ContainerStarted","Data":"2463de37b2f82ffdea8aeede5cad7d22c54549e92c419bdb5810450eca93a353"}
Sep 12 01:55:50.233263 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:50.233215    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh" podStartSLOduration=2.024781663 podStartE2EDuration="2.233197859s" totalImagesPullingTime="208.416196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:55:50.231461044 +0000 UTC m=+644.158406932" watchObservedRunningTime="2026-09-12 01:55:50.233197859 +0000 UTC m=+644.160143778"
Sep 12 01:55:59.137885 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:59.137851    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:59.138325 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:59.138137    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:59.150572 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:59.150551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:55:59.262377 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:55:59.262350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-d89mh"
Sep 12 01:56:29.719652 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.719620    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"]
Sep 12 01:56:29.724547 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.724525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.727100 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.727080    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 12 01:56:29.734248 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.734226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"]
Sep 12 01:56:29.841143 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.841292 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841148    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.841292 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841175    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbrl\" (UniqueName: \"kubernetes.io/projected/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-kube-api-access-slbrl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.841292 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841225    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.841292 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841260    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.841432 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.841360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.942794 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.942761    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.942981 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.942799    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.942981 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.942825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943141 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbrl\" (UniqueName: \"kubernetes.io/projected/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-kube-api-access-slbrl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943141 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943141 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943348 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943458 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.943574 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.943489    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.945271 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.945250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.945683 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.945663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:29.951266 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:29.951241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbrl\" (UniqueName: \"kubernetes.io/projected/38956ee2-20d5-4eaa-afcf-6a2913ae22ff-kube-api-access-slbrl\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77\" (UID: \"38956ee2-20d5-4eaa-afcf-6a2913ae22ff\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:30.035939 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:30.035892    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:30.304928 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:30.304905    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"]
Sep 12 01:56:30.307697 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:56:30.307671    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38956ee2_20d5_4eaa_afcf_6a2913ae22ff.slice/crio-ad5b813f53678f1f0060a210fa44edcbbdc2aa5833a25fcd22d7856a25df2eee WatchSource:0}: Error finding container ad5b813f53678f1f0060a210fa44edcbbdc2aa5833a25fcd22d7856a25df2eee: Status 404 returned error can't find the container with id ad5b813f53678f1f0060a210fa44edcbbdc2aa5833a25fcd22d7856a25df2eee
Sep 12 01:56:30.382003 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:30.381972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77" event={"ID":"38956ee2-20d5-4eaa-afcf-6a2913ae22ff","Type":"ContainerStarted","Data":"ad5b813f53678f1f0060a210fa44edcbbdc2aa5833a25fcd22d7856a25df2eee"}
Sep 12 01:56:31.387423 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:31.387389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77" event={"ID":"38956ee2-20d5-4eaa-afcf-6a2913ae22ff","Type":"ContainerStarted","Data":"5454b9d409d899031902d1c533759736ba89f78bf6b43f232b6dc3cc2787110d"}
Sep 12 01:56:31.409368 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:31.409322    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77" podStartSLOduration=2.245041129 podStartE2EDuration="2.40930674s" totalImagesPullingTime="164.265611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:56:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:56:31.40848944 +0000 UTC m=+685.335435327" watchObservedRunningTime="2026-09-12 01:56:31.40930674 +0000 UTC m=+685.336252626"
Sep 12 01:56:40.036019 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:40.035988    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:40.036404 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:40.036088    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:40.048508 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:40.048484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:56:40.438533 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:56:40.438505    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77"
Sep 12 01:58:03.506974 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.506942    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 01:58:03.510302 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.510280    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.512976 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.512952    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3fc028-openshift-default-dockercfg-l2qgh\""
Sep 12 01:58:03.522473 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.522450    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 01:58:03.595691 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.595691 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.595939 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595731    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.595939 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.595939 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595862    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.596134 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.595944    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85xkn\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.596134 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.596029    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.596134 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.596109    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.596288 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.596173    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.651334 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.651302    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 01:58:03.654741 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.654716    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.662844 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.662815    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f660a6-openshift-default-dockercfg-h425j\""
Sep 12 01:58:03.671560 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.671539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 01:58:03.697090 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697235 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697235 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhpj8\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697355 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697232    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697355 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697323    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697469 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697469 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697469 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697621 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697482    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697621 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697621 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697563    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697778 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697643    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697778 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697887 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.697887 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697839    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85xkn\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.697887 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697871    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.698084 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.698084 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.697961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.699121 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.699094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.699891 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.699868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.700209 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.700186    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.700209 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.700173    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.700356 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.700010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.702807 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.702783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.703511 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.703488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.707910 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.707869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85xkn\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.708112 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.708024    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token\") pod \"e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") " pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.799149 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhpj8\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799149 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799168    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799189    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799285    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799367 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799755 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799639    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799755 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799707    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.799931 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799907    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.800019 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.799966    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.800358 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.800336    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.801602 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.801576    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.801909 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.801890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.807248 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.807219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhpj8\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.807833 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.807817    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token\") pod \"e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") " pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:03.821829 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.821804    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:03.966590 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:03.966559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:04.105776 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.105752    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 01:58:04.108313 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:04.108286    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a28a19b_9b97_4110_8bb6_a5125ae1bcc3.slice/crio-160e498437b736c337c876e4e805da3d68a0bf46100342f12e163cfab552ac41 WatchSource:0}: Error finding container 160e498437b736c337c876e4e805da3d68a0bf46100342f12e163cfab552ac41: Status 404 returned error can't find the container with id 160e498437b736c337c876e4e805da3d68a0bf46100342f12e163cfab552ac41
Sep 12 01:58:04.110528 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.110493    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.110628 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.110558    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.110628 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.110591    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.249599 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.249507    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 01:58:04.252754 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:04.252714    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633d9bc5_d002_488f_80c6_c9f974b0de35.slice/crio-2104784e2c076a5213b49f2f40a86fe2e039cd9756f2b62e80e56d382564ec70 WatchSource:0}: Error finding container 2104784e2c076a5213b49f2f40a86fe2e039cd9756f2b62e80e56d382564ec70: Status 404 returned error can't find the container with id 2104784e2c076a5213b49f2f40a86fe2e039cd9756f2b62e80e56d382564ec70
Sep 12 01:58:04.255364 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.255318    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.255440 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.255414    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.255478 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.255457    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:04.783603 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.783561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerStarted","Data":"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"}
Sep 12 01:58:04.784014 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.783627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerStarted","Data":"2104784e2c076a5213b49f2f40a86fe2e039cd9756f2b62e80e56d382564ec70"}
Sep 12 01:58:04.785236 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.785209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerStarted","Data":"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"}
Sep 12 01:58:04.785345 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.785256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerStarted","Data":"160e498437b736c337c876e4e805da3d68a0bf46100342f12e163cfab552ac41"}
Sep 12 01:58:04.822589 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.822567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:04.837590 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.837526    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podStartSLOduration=1.837505625 podStartE2EDuration="1.837505625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:04.835986941 +0000 UTC m=+778.762932827" watchObservedRunningTime="2026-09-12 01:58:04.837505625 +0000 UTC m=+778.764451512"
Sep 12 01:58:04.868162 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.868118    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podStartSLOduration=1.868102092 podStartE2EDuration="1.868102092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:04.866258821 +0000 UTC m=+778.793204707" watchObservedRunningTime="2026-09-12 01:58:04.868102092 +0000 UTC m=+778.795047977"
Sep 12 01:58:04.967311 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:04.967282    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:05.828027 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:05.828001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:05.973067 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:05.973014    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:06.795402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:06.795273    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:06.795402 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:06.795335    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:06.796180 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:06.796159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:06.796317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:06.796218    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:08.374916 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.374881    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 01:58:08.382773 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.382731    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 01:58:08.382933 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.382850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.385952 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.385850    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-f660a6-serving-cert\""
Sep 12 01:58:08.386377 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.386138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-565nn\""
Sep 12 01:58:08.443381 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.443334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.443539 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.443454    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz5fd\" (UniqueName: \"kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.443539 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.443531    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.443651 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.443566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.544922 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.544891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz5fd\" (UniqueName: \"kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.545105 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.544942    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.545105 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.544966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.545105 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.545027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.545634 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.545604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.547502 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.547481    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.547657 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.547636    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.556126 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.556102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz5fd\" (UniqueName: \"kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd\") pod \"maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.697392 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.697302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:08.977388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:08.977361    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 01:58:08.980022 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:08.979979    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod152472a3_44b0_4843_ba42_2ee526496455.slice/crio-d20e64fe2e5d7a5376152a65e8c4f546db072c198fea325360b6234ac77bb2b2 WatchSource:0}: Error finding container d20e64fe2e5d7a5376152a65e8c4f546db072c198fea325360b6234ac77bb2b2: Status 404 returned error can't find the container with id d20e64fe2e5d7a5376152a65e8c4f546db072c198fea325360b6234ac77bb2b2
Sep 12 01:58:09.119372 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.119329    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 01:58:09.124063 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.124010    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.126719 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.126692    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-f660a6\""
Sep 12 01:58:09.127067 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.127027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-f660a6-dockercfg-4llpm\""
Sep 12 01:58:09.135817 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.135795    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 01:58:09.151919 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.151893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.152089 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.152069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm64j\" (UniqueName: \"kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.252616 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.252525    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm64j\" (UniqueName: \"kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.252616 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.252570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.253317 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.253284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.261739 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.261711    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm64j\" (UniqueName: \"kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j\") pod \"payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.438479 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.438456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.731751 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.731720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 01:58:09.735581 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:09.735551    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0770b0_8ad1_46fe_8592_ebc1dfecf4aa.slice/crio-20524fc8a643e0c16dde537e41e6e16560a89ddff8baaa95b7095f5ad265bc21 WatchSource:0}: Error finding container 20524fc8a643e0c16dde537e41e6e16560a89ddff8baaa95b7095f5ad265bc21: Status 404 returned error can't find the container with id 20524fc8a643e0c16dde537e41e6e16560a89ddff8baaa95b7095f5ad265bc21
Sep 12 01:58:09.813519 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.813490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" event={"ID":"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa","Type":"ContainerStarted","Data":"5c0bf5230a5b5c4b795895f333fc5d0f272de4ab318df3e457f733deebbbfe6e"}
Sep 12 01:58:09.813646 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.813548    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" event={"ID":"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa","Type":"ContainerStarted","Data":"20524fc8a643e0c16dde537e41e6e16560a89ddff8baaa95b7095f5ad265bc21"}
Sep 12 01:58:09.813646 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.813578    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:09.815335 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.815311    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" event={"ID":"152472a3-44b0-4843-ba42-2ee526496455","Type":"ContainerStarted","Data":"3390552ff99c0d3fbc9c5d2a2920d1e4764eaeb7d3d400b9c28716257a65121f"}
Sep 12 01:58:09.815441 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.815352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" event={"ID":"152472a3-44b0-4843-ba42-2ee526496455","Type":"ContainerStarted","Data":"d20e64fe2e5d7a5376152a65e8c4f546db072c198fea325360b6234ac77bb2b2"}
Sep 12 01:58:09.815441 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.815386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:09.837160 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.837117    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" podStartSLOduration=0.837090184 podStartE2EDuration="837.090184ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:09.836533253 +0000 UTC m=+783.763479142" watchObservedRunningTime="2026-09-12 01:58:09.837090184 +0000 UTC m=+783.764036071"
Sep 12 01:58:09.865068 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:09.865000    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" podStartSLOduration=1.421231324 podStartE2EDuration="1.864980148s" totalImagesPullingTime="443.748824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:09.863131485 +0000 UTC m=+783.790077374" watchObservedRunningTime="2026-09-12 01:58:09.864980148 +0000 UTC m=+783.791926036"
Sep 12 01:58:10.638954 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.638919    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 01:58:10.642982 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.642953    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.645509 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.645479    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-98f68e-serving-cert\""
Sep 12 01:58:10.659208 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.659182    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 01:58:10.768227 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.768165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncz94\" (UniqueName: \"kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.768400 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.768292    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.768400 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.768346    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.768501 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.768430    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.869463 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.869420    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.869463 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.869465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.869722 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:58:10.869575    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-serving-cert: secret "maas-api-e2e-worker-w4-98f68e-serving-cert" not found
Sep 12 01:58:10.869722 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:58:10.869641    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls podName:ca9fc4ef-79c2-4016-881d-599a91f818e1 nodeName:}" failed. No retries permitted until 2026-09-12 01:58:11.36962157 +0000 UTC m=+785.296567434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls") pod "maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" (UID: "ca9fc4ef-79c2-4016-881d-599a91f818e1") : secret "maas-api-e2e-worker-w4-98f68e-serving-cert" not found
Sep 12 01:58:10.869839 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.869751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.869839 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.869816    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncz94\" (UniqueName: \"kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.870607 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.870581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.872215 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.872193    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:10.879738 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:10.879717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncz94\" (UniqueName: \"kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:11.375489 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.375445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:11.379593 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.379512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") pod \"maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:11.557211 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.557181    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:11.633918 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.633850    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 01:58:11.643491 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.643458    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.646710 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.646682    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-98f68e\""
Sep 12 01:58:11.647032 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.647012    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-98f68e-dockercfg-k9ddg\""
Sep 12 01:58:11.649700 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.649680    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 01:58:11.678712 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.678690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.678812 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.678759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwhd\" (UniqueName: \"kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.779441 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.779393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.779621 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.779514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwhd\" (UniqueName: \"kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.780253 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.780228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.788513 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.788488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwhd\" (UniqueName: \"kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd\") pod \"payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:11.830458 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.830437    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 01:58:11.832294 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:11.832261    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca9fc4ef_79c2_4016_881d_599a91f818e1.slice/crio-6681326bc41f09010cdf156d4779cdfe35f336a41614ddfd13200ce9e06dfb8c WatchSource:0}: Error finding container 6681326bc41f09010cdf156d4779cdfe35f336a41614ddfd13200ce9e06dfb8c: Status 404 returned error can't find the container with id 6681326bc41f09010cdf156d4779cdfe35f336a41614ddfd13200ce9e06dfb8c
Sep 12 01:58:11.957247 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:11.957168    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:12.229809 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.229739    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 01:58:12.233113 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:12.233012    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec9c7d8_8e68_42f5_953a_07488d2b7504.slice/crio-80e8bb32b24abf0ce593ad04f84d1e8138f077089489cdf57257d2062b6f058d WatchSource:0}: Error finding container 80e8bb32b24abf0ce593ad04f84d1e8138f077089489cdf57257d2062b6f058d: Status 404 returned error can't find the container with id 80e8bb32b24abf0ce593ad04f84d1e8138f077089489cdf57257d2062b6f058d
Sep 12 01:58:12.832777 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.832743    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" event={"ID":"ca9fc4ef-79c2-4016-881d-599a91f818e1","Type":"ContainerStarted","Data":"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"}
Sep 12 01:58:12.833253 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.832803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" event={"ID":"ca9fc4ef-79c2-4016-881d-599a91f818e1","Type":"ContainerStarted","Data":"6681326bc41f09010cdf156d4779cdfe35f336a41614ddfd13200ce9e06dfb8c"}
Sep 12 01:58:12.833253 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.832830    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:12.838023 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.837999    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" event={"ID":"eec9c7d8-8e68-42f5-953a-07488d2b7504","Type":"ContainerStarted","Data":"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"}
Sep 12 01:58:12.838163 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.838073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" event={"ID":"eec9c7d8-8e68-42f5-953a-07488d2b7504","Type":"ContainerStarted","Data":"80e8bb32b24abf0ce593ad04f84d1e8138f077089489cdf57257d2062b6f058d"}
Sep 12 01:58:12.838163 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.838103    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:12.852492 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.852437    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" podStartSLOduration=2.557582452 podStartE2EDuration="2.852421388s" totalImagesPullingTime="294.838936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:12.851946143 +0000 UTC m=+786.778892028" watchObservedRunningTime="2026-09-12 01:58:12.852421388 +0000 UTC m=+786.779367275"
Sep 12 01:58:12.873095 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:12.873034    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" podStartSLOduration=1.873019585 podStartE2EDuration="1.873019585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:12.872549573 +0000 UTC m=+786.799495460" watchObservedRunningTime="2026-09-12 01:58:12.873019585 +0000 UTC m=+786.799965465"
Sep 12 01:58:15.833654 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:15.833625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 01:58:18.849206 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:18.849178    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 01:58:20.594140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.594099    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 01:58:20.600609 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.600363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 01:58:20.600609 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.600506    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.603622 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.603595    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-9ef13e-dockercfg-vk9q9\""
Sep 12 01:58:20.603848 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.603830    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-9ef13e\""
Sep 12 01:58:20.687684 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.687637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.688097 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.688073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v77xt\" (UniqueName: \"kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.788831 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.788781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v77xt\" (UniqueName: \"kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.789013 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.788901    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.789879 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.789845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.798836 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.798803    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v77xt\" (UniqueName: \"kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt\") pod \"payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") " pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:20.822907 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.822883    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 01:58:20.912077 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:20.912028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:21.200370 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.200344    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 01:58:21.202760 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:21.202727    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e502a32_f15a_4764_92fa_56f1bdcf0ca4.slice/crio-fbc11154734df3bdeadc4c6a293ab38efa5c7cec18d518dc39985e0ed8eb39d0 WatchSource:0}: Error finding container fbc11154734df3bdeadc4c6a293ab38efa5c7cec18d518dc39985e0ed8eb39d0: Status 404 returned error can't find the container with id fbc11154734df3bdeadc4c6a293ab38efa5c7cec18d518dc39985e0ed8eb39d0
Sep 12 01:58:21.806237 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.806140    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:21.810237 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.810215    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:21.814474 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.814288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-57c4f4a4-dockercfg-vthgv\""
Sep 12 01:58:21.814474 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.814322    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-57c4f4a4\""
Sep 12 01:58:21.818528 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.818487    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:21.894407 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.894361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" event={"ID":"9e502a32-f15a-4764-92fa-56f1bdcf0ca4","Type":"ContainerStarted","Data":"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"}
Sep 12 01:58:21.894584 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.894422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" event={"ID":"9e502a32-f15a-4764-92fa-56f1bdcf0ca4","Type":"ContainerStarted","Data":"fbc11154734df3bdeadc4c6a293ab38efa5c7cec18d518dc39985e0ed8eb39d0"}
Sep 12 01:58:21.894584 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.894496    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:21.901529 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.901499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrcj\" (UniqueName: \"kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:21.901672 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.901618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:21.924439 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:21.924394    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" podStartSLOduration=1.924381025 podStartE2EDuration="1.924381025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:21.922616768 +0000 UTC m=+795.849562654" watchObservedRunningTime="2026-09-12 01:58:21.924381025 +0000 UTC m=+795.851326910"
Sep 12 01:58:22.002282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.002236    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.002445 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.002360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrcj\" (UniqueName: \"kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.003161 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.003134    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.034692 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.034655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrcj\" (UniqueName: \"kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj\") pod \"payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") " pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.129906 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.129867    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.420709 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.420676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:22.423303 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:22.423240    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eeba395_cecb_4ba2_870f_f337ff8d6c53.slice/crio-2bd81a432c03dce59a9095e2b58612d892e8e7ebc7c8021b957824e113e2e3f5 WatchSource:0}: Error finding container 2bd81a432c03dce59a9095e2b58612d892e8e7ebc7c8021b957824e113e2e3f5: Status 404 returned error can't find the container with id 2bd81a432c03dce59a9095e2b58612d892e8e7ebc7c8021b957824e113e2e3f5
Sep 12 01:58:22.837272 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.837177    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 01:58:22.837272 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.837232    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 01:58:22.837794 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.837239    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 01:58:22.837794 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.837293    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 01:58:22.904645 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.904517    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" event={"ID":"9eeba395-cecb-4ba2-870f-f337ff8d6c53","Type":"ContainerStarted","Data":"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"}
Sep 12 01:58:22.904645 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.904588    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:22.904645 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.904613    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" event={"ID":"9eeba395-cecb-4ba2-870f-f337ff8d6c53","Type":"ContainerStarted","Data":"2bd81a432c03dce59a9095e2b58612d892e8e7ebc7c8021b957824e113e2e3f5"}
Sep 12 01:58:22.926051 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:22.925980    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" podStartSLOduration=1.9259591230000002 podStartE2EDuration="1.925959123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:22.922754341 +0000 UTC m=+796.849700228" watchObservedRunningTime="2026-09-12 01:58:22.925959123 +0000 UTC m=+796.852905009"
Sep 12 01:58:23.845969 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.845878    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 01:58:23.913233 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.913195    2582 generic.go:356] "Generic (PLEG): container finished" podID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerID="05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588" exitCode=0
Sep 12 01:58:23.913406 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.913279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerDied","Data":"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"}
Sep 12 01:58:23.914027 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.913974    2582 scope.go:117] "RemoveContainer" containerID="05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"
Sep 12 01:58:23.915313 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.915282    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:23.915419 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.915351    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:23.915419 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:23.915382    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:24.922692 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.922657    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerID="74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4" exitCode=0
Sep 12 01:58:24.923452 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.923426    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerDied","Data":"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"}
Sep 12 01:58:24.924108 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.924089    2582 scope.go:117] "RemoveContainer" containerID="74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"
Sep 12 01:58:24.928157 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.927625    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:24.928157 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.927838    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:24.928157 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.927892    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 01:58:24.932077 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.929934    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerStarted","Data":"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"}
Sep 12 01:58:24.967755 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.967726    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:24.967934 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.967912    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:24.968013 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:24.967967    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:25.936936 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:25.936903    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerStarted","Data":"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"}
Sep 12 01:58:25.967855 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:25.967822    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:25.968011 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:25.967868    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:26.823166 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:26.823136    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:26.823404 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:26.823277    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 12 01:58:26.823522 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:26.823505    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 12 01:58:26.966889 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:26.966859    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:26.967357 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:26.966914    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:27.824905 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:27.824442    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 12 01:58:27.824905 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:27.824515    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 12 01:58:27.967333 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:27.967296    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:27.967710 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:27.967352    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:28.822822 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:28.822776    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 12 01:58:28.823027 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:28.822829    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 12 01:58:28.967960 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:28.967922    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:28.968352 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:28.967980    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:29.822496 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:29.822459    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 12 01:58:29.822778 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:29.822745    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 12 01:58:29.967506 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:29.967476    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Sep 12 01:58:29.967720 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:29.967701    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Sep 12 01:58:30.822605 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:30.822568    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 12 01:58:30.822980 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:30.822640    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 12 01:58:30.974541 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:30.974510    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:30.974943 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:30.974918    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:30.975969 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:30.975945    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 01:58:31.826321 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.826287    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:31.826822 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.826562    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:31.827215 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.827199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 01:58:31.860415 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.860386    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 01:58:31.864112 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.864091    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:31.866780 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.866760    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7d6e9f-serving-cert\""
Sep 12 01:58:31.876278 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:31.876255    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 01:58:32.014780 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.014745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.014932 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.014785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrps6\" (UniqueName: \"kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.014932 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.014881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.015011 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.014974    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.116061 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.115949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.116233 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.116066    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrps6\" (UniqueName: \"kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.116233 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.116143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.116370 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.116342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.117016 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.116995    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.118500 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.118476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.118677 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.118654    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.128627 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.128605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrps6\" (UniqueName: \"kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6\") pod \"maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.175086 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.175056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:32.456251 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.456224    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 01:58:32.458196 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:32.458167    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31de3ece_f720_42b9_aeca_e6a306e1b249.slice/crio-0efd0abd4c79f45a1d6cc0aea74f7c1ecae6ec6eed030efcf60fd7d9dcdfa6aa WatchSource:0}: Error finding container 0efd0abd4c79f45a1d6cc0aea74f7c1ecae6ec6eed030efcf60fd7d9dcdfa6aa: Status 404 returned error can't find the container with id 0efd0abd4c79f45a1d6cc0aea74f7c1ecae6ec6eed030efcf60fd7d9dcdfa6aa
Sep 12 01:58:32.563651 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.563622    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 01:58:32.566124 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.566100    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.568774 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.568755    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7d6e9f-dockercfg-gvcvf\""
Sep 12 01:58:32.568878 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.568781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7d6e9f\""
Sep 12 01:58:32.580779 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.580754    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 01:58:32.723478 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.723410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.723624 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.723485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzm6\" (UniqueName: \"kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.824981 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.824951    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.825140 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.825057    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzm6\" (UniqueName: \"kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.825690 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.825670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.835099 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.835077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzm6\" (UniqueName: \"kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6\") pod \"payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.881175 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.881151    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:32.906331 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.906314    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 01:58:32.976643 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.975863    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" event={"ID":"31de3ece-f720-42b9-aeca-e6a306e1b249","Type":"ContainerStarted","Data":"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"}
Sep 12 01:58:32.976643 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.975960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" event={"ID":"31de3ece-f720-42b9-aeca-e6a306e1b249","Type":"ContainerStarted","Data":"0efd0abd4c79f45a1d6cc0aea74f7c1ecae6ec6eed030efcf60fd7d9dcdfa6aa"}
Sep 12 01:58:32.976643 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:32.976514    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:33.001118 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.001059    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" podStartSLOduration=1.728849635 podStartE2EDuration="2.001024461s" totalImagesPullingTime="272.174826ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:32.998450539 +0000 UTC m=+806.925396426" watchObservedRunningTime="2026-09-12 01:58:33.001024461 +0000 UTC m=+806.927970349"
Sep 12 01:58:33.154600 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.154576    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 01:58:33.155961 ip-10-0-128-81 kubenswrapper[2582]: W0912 01:58:33.155937    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad517be6_9b16_44eb_a5a9_2a1b8283282f.slice/crio-a324f77843c17f24290b9592b3c82e11ccf5c542314dcdf153790b7e14cbc515 WatchSource:0}: Error finding container a324f77843c17f24290b9592b3c82e11ccf5c542314dcdf153790b7e14cbc515: Status 404 returned error can't find the container with id a324f77843c17f24290b9592b3c82e11ccf5c542314dcdf153790b7e14cbc515
Sep 12 01:58:33.916420 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.916390    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:33.984114 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.984066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" event={"ID":"ad517be6-9b16-44eb-a5a9-2a1b8283282f","Type":"ContainerStarted","Data":"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"}
Sep 12 01:58:33.984282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.984131    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" event={"ID":"ad517be6-9b16-44eb-a5a9-2a1b8283282f","Type":"ContainerStarted","Data":"a324f77843c17f24290b9592b3c82e11ccf5c542314dcdf153790b7e14cbc515"}
Sep 12 01:58:33.984282 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:33.984225    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:34.000246 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:34.000199    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" podStartSLOduration=2.000184821 podStartE2EDuration="2.000184821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 01:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 01:58:33.999556356 +0000 UTC m=+807.926502247" watchObservedRunningTime="2026-09-12 01:58:34.000184821 +0000 UTC m=+807.927130708"
Sep 12 01:58:38.989840 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:38.989808    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 01:58:44.992434 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:44.992404    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 01:58:46.636851 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.636813    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:46.637345 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.637099    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" podUID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" containerName="payload-processing" containerID="cri-o://d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b" gracePeriod=30
Sep 12 01:58:46.890722 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.890662    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:46.943121 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.943090    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gzrcj\" (UniqueName: \"kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj\") pod \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") "
Sep 12 01:58:46.943279 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.943154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume\") pod \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\" (UID: \"9eeba395-cecb-4ba2-870f-f337ff8d6c53\") "
Sep 12 01:58:46.943530 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.943507    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9eeba395-cecb-4ba2-870f-f337ff8d6c53" (UID: "9eeba395-cecb-4ba2-870f-f337ff8d6c53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 01:58:46.945308 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:46.945272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj" (OuterVolumeSpecName: "kube-api-access-gzrcj") pod "9eeba395-cecb-4ba2-870f-f337ff8d6c53" (UID: "9eeba395-cecb-4ba2-870f-f337ff8d6c53"). InnerVolumeSpecName "kube-api-access-gzrcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 01:58:47.044549 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.044508    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eeba395-cecb-4ba2-870f-f337ff8d6c53-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:58:47.044549 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.044541    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrcj\" (UniqueName: \"kubernetes.io/projected/9eeba395-cecb-4ba2-870f-f337ff8d6c53-kube-api-access-gzrcj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 01:58:47.050236 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.050204    2582 generic.go:356] "Generic (PLEG): container finished" podID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" containerID="d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b" exitCode=0
Sep 12 01:58:47.050388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.050260    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"
Sep 12 01:58:47.050388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.050295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" event={"ID":"9eeba395-cecb-4ba2-870f-f337ff8d6c53","Type":"ContainerDied","Data":"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"}
Sep 12 01:58:47.050388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.050344    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck" event={"ID":"9eeba395-cecb-4ba2-870f-f337ff8d6c53","Type":"ContainerDied","Data":"2bd81a432c03dce59a9095e2b58612d892e8e7ebc7c8021b957824e113e2e3f5"}
Sep 12 01:58:47.050388 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.050366    2582 scope.go:117] "RemoveContainer" containerID="d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"
Sep 12 01:58:47.060509 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.060491    2582 scope.go:117] "RemoveContainer" containerID="d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"
Sep 12 01:58:47.060799 ip-10-0-128-81 kubenswrapper[2582]: E0912 01:58:47.060773    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b\": container with ID starting with d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b not found: ID does not exist" containerID="d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"
Sep 12 01:58:47.060847 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.060810    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b"} err="failed to get container status \"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b\": rpc error: code = NotFound desc = could not find container \"d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b\": container with ID starting with d1d1dadf0bbb2cd459002e5a8aaff085acf012fe4d06a53aba0d5ba6e4f08d5b not found: ID does not exist"
Sep 12 01:58:47.074316 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.074294    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:47.079265 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:47.079244    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-57c4f4a4-78d9d79679-wggck"]
Sep 12 01:58:48.555571 ip-10-0-128-81 kubenswrapper[2582]: I0912 01:58:48.555542    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" path="/var/lib/kubelet/pods/9eeba395-cecb-4ba2-870f-f337ff8d6c53/volumes"
Sep 12 02:00:00.176979 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.176892    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"]
Sep 12 02:00:00.177534 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.177505    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" containerName="payload-processing"
Sep 12 02:00:00.177534 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.177529    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" containerName="payload-processing"
Sep 12 02:00:00.177669 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.177631    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eeba395-cecb-4ba2-870f-f337ff8d6c53" containerName="payload-processing"
Sep 12 02:00:00.180735 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.180712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"
Sep 12 02:00:00.183221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.183193    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x88s\""
Sep 12 02:00:00.186574 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.186550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"]
Sep 12 02:00:00.207952 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.207928    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"
Sep 12 02:00:00.482088 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:00.482062    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"]
Sep 12 02:00:00.484005 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:00:00.483970    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8d24fa_8c7f_4dd0_b165_65acd2641b89.slice/crio-25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5 WatchSource:0}: Error finding container 25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5: Status 404 returned error can't find the container with id 25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5
Sep 12 02:00:01.416875 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:01.416835    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm" event={"ID":"5a8d24fa-8c7f-4dd0-b165-65acd2641b89","Type":"ContainerStarted","Data":"25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5"}
Sep 12 02:00:04.435475 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:04.435378    2582 generic.go:356] "Generic (PLEG): container finished" podID="5a8d24fa-8c7f-4dd0-b165-65acd2641b89" containerID="eff1ff2f989e84367f38cd4ae74dbc8621a0e8d16a9bdb320eb1093f1856b9ac" exitCode=0
Sep 12 02:00:04.435475 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:04.435421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm" event={"ID":"5a8d24fa-8c7f-4dd0-b165-65acd2641b89","Type":"ContainerDied","Data":"eff1ff2f989e84367f38cd4ae74dbc8621a0e8d16a9bdb320eb1093f1856b9ac"}
Sep 12 02:00:05.575756 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:05.575730    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"
Sep 12 02:00:06.448309 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:06.448277    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"
Sep 12 02:00:06.448309 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:06.448296    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm" event={"ID":"5a8d24fa-8c7f-4dd0-b165-65acd2641b89","Type":"ContainerDied","Data":"25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5"}
Sep 12 02:00:06.448614 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:06.448379    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25e716f72d69e1327efd45812a615d7d2ea37ee2704f3e8b003d496767e3c4c5"
Sep 12 02:00:06.540226 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:06.540197    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:00:06.540398 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:06.540197    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:00:56.767243 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:56.767209    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 02:00:56.767767 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:56.767464    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" podUID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" containerName="payload-processing" containerID="cri-o://46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0" gracePeriod=30
Sep 12 02:00:57.043209 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.043186    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 02:00:57.196999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.196966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume\") pod \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") "
Sep 12 02:00:57.197213 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.197012    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v77xt\" (UniqueName: \"kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt\") pod \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\" (UID: \"9e502a32-f15a-4764-92fa-56f1bdcf0ca4\") "
Sep 12 02:00:57.197439 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.197407    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e502a32-f15a-4764-92fa-56f1bdcf0ca4" (UID: "9e502a32-f15a-4764-92fa-56f1bdcf0ca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:00:57.199227 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.199202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt" (OuterVolumeSpecName: "kube-api-access-v77xt") pod "9e502a32-f15a-4764-92fa-56f1bdcf0ca4" (UID: "9e502a32-f15a-4764-92fa-56f1bdcf0ca4"). InnerVolumeSpecName "kube-api-access-v77xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:00:57.298094 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.297958    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:00:57.298094 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.297992    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v77xt\" (UniqueName: \"kubernetes.io/projected/9e502a32-f15a-4764-92fa-56f1bdcf0ca4-kube-api-access-v77xt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:00:57.705950 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.705918    2582 generic.go:356] "Generic (PLEG): container finished" podID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" containerID="46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0" exitCode=0
Sep 12 02:00:57.706134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.705959    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"
Sep 12 02:00:57.706134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.705980    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" event={"ID":"9e502a32-f15a-4764-92fa-56f1bdcf0ca4","Type":"ContainerDied","Data":"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"}
Sep 12 02:00:57.706134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.706022    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th" event={"ID":"9e502a32-f15a-4764-92fa-56f1bdcf0ca4","Type":"ContainerDied","Data":"fbc11154734df3bdeadc4c6a293ab38efa5c7cec18d518dc39985e0ed8eb39d0"}
Sep 12 02:00:57.706134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.706067    2582 scope.go:117] "RemoveContainer" containerID="46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"
Sep 12 02:00:57.716051 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.716018    2582 scope.go:117] "RemoveContainer" containerID="46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"
Sep 12 02:00:57.716338 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:00:57.716316    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0\": container with ID starting with 46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0 not found: ID does not exist" containerID="46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"
Sep 12 02:00:57.716409 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.716352    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0"} err="failed to get container status \"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0\": rpc error: code = NotFound desc = could not find container \"46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0\": container with ID starting with 46a741ed3b75659ea5b3f8117ed8faa1714e6918e52d618846101bb93a4663c0 not found: ID does not exist"
Sep 12 02:00:57.727523 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.727498    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 02:00:57.732271 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:57.732249    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-9ef13e-7f466b5559-sw6th"]
Sep 12 02:00:58.555526 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:00:58.555498    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" path="/var/lib/kubelet/pods/9e502a32-f15a-4764-92fa-56f1bdcf0ca4/volumes"
Sep 12 02:02:31.111958 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.111922    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:02:31.112683 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112658    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8d24fa-8c7f-4dd0-b165-65acd2641b89" containerName="cleanup"
Sep 12 02:02:31.112683 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8d24fa-8c7f-4dd0-b165-65acd2641b89" containerName="cleanup"
Sep 12 02:02:31.112830 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112697    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" containerName="payload-processing"
Sep 12 02:02:31.112830 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112705    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" containerName="payload-processing"
Sep 12 02:02:31.112830 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112804    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e502a32-f15a-4764-92fa-56f1bdcf0ca4" containerName="payload-processing"
Sep 12 02:02:31.112830 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.112818    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8d24fa-8c7f-4dd0-b165-65acd2641b89" containerName="cleanup"
Sep 12 02:02:31.116056 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.116025    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.118679 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.118658    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d5764ff8-openshift-default-dockercfg-gljhf\""
Sep 12 02:02:31.127275 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.127250    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:02:31.214730 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214690    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214730 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.214947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.214922    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9x8c\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.215220 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.215022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.215220 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.215105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316473 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316650 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316650 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316650 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316650 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316650 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316676    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316711    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.316926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316733    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9x8c\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.317101 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.316988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.317101 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.317013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.317198 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.317098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.317198 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.317126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.317473 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.317449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.318885 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.318863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.319251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.319232    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.325237 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.325215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.325440 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.325417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9x8c\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c\") pod \"e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") " pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.427558 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.427528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:31.716310 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.716283    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:02:31.718552 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:02:31.718520    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97ce6e1_0661_4fed_9ce1_8ffe6e940379.slice/crio-d4f52eabe21806e5840042f722f7c94908a66f3c8fed77e0afa1f33c5f89cb3c WatchSource:0}: Error finding container d4f52eabe21806e5840042f722f7c94908a66f3c8fed77e0afa1f33c5f89cb3c: Status 404 returned error can't find the container with id d4f52eabe21806e5840042f722f7c94908a66f3c8fed77e0afa1f33c5f89cb3c
Sep 12 02:02:31.721626 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.721594    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:02:31.721694 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.721661    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:02:31.721734 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:31.721690    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:02:32.168252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:32.168217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerStarted","Data":"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"}
Sep 12 02:02:32.168689 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:32.168267    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerStarted","Data":"d4f52eabe21806e5840042f722f7c94908a66f3c8fed77e0afa1f33c5f89cb3c"}
Sep 12 02:02:32.188516 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:32.188458    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podStartSLOduration=1.188441862 podStartE2EDuration="1.188441862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:02:32.186281338 +0000 UTC m=+1046.113227224" watchObservedRunningTime="2026-09-12 02:02:32.188441862 +0000 UTC m=+1046.115387748"
Sep 12 02:02:32.428759 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:32.428679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:32.433663 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:32.433642    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:33.173072 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:33.173019    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:33.173940 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:33.173921    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:02:40.621848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.621817    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:02:40.625414 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.625391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.628159 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.628137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d5764ff8-serving-cert\""
Sep 12 02:02:40.638942 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.638892    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:02:40.709715 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.709683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.709880 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.709739    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.709880 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.709825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnkj\" (UniqueName: \"kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.709880 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.709864    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.811218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.811182    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.811397 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.811233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.811397 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.811276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnkj\" (UniqueName: \"kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.811397 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.811302    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.811539 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:02:40.811414    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-serving-cert: secret "maas-api-e2e-mt-d5764ff8-serving-cert" not found
Sep 12 02:02:40.811539 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:02:40.811502    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls podName:f22e336c-6c51-4e73-a231-5bf284d1be75 nodeName:}" failed. No retries permitted until 2026-09-12 02:02:41.31147906 +0000 UTC m=+1055.238424937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls") pod "maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" (UID: "f22e336c-6c51-4e73-a231-5bf284d1be75") : secret "maas-api-e2e-mt-d5764ff8-serving-cert" not found
Sep 12 02:02:40.811923 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.811901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.813700 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.813678    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:40.821843 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:40.821819    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnkj\" (UniqueName: \"kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:41.319183 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:41.317476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:41.322370 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:41.321059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") pod \"maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:41.537347 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:41.537312    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:41.808588 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:41.808551    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:02:41.810236 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:02:41.810209    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22e336c_6c51_4e73_a231_5bf284d1be75.slice/crio-96487fe200be2cf9f1d5004b9b0e27265d3d63dd6ef657f8071d76eba81d7031 WatchSource:0}: Error finding container 96487fe200be2cf9f1d5004b9b0e27265d3d63dd6ef657f8071d76eba81d7031: Status 404 returned error can't find the container with id 96487fe200be2cf9f1d5004b9b0e27265d3d63dd6ef657f8071d76eba81d7031
Sep 12 02:02:41.811601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:41.811581    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:02:42.217769 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:42.217739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" event={"ID":"f22e336c-6c51-4e73-a231-5bf284d1be75","Type":"ContainerStarted","Data":"96487fe200be2cf9f1d5004b9b0e27265d3d63dd6ef657f8071d76eba81d7031"}
Sep 12 02:02:43.225318 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:43.225286    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" event={"ID":"f22e336c-6c51-4e73-a231-5bf284d1be75","Type":"ContainerStarted","Data":"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"}
Sep 12 02:02:43.225741 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:43.225433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:49.235210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:49.235180    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:02:49.255458 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:02:49.254896    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" podStartSLOduration=8.8850045 podStartE2EDuration="9.254877695s" totalImagesPullingTime="369.873195ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:02:43.24243193 +0000 UTC m=+1057.169377817" watchObservedRunningTime="2026-09-12 02:02:49.254877695 +0000 UTC m=+1063.181823582"
Sep 12 02:03:04.214196 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:04.214158    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 02:03:04.214603 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:04.214222    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 02:03:05.342576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.342552    2582 generic.go:356] "Generic (PLEG): container finished" podID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerID="0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880" exitCode=0
Sep 12 02:03:05.342907 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.342624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerDied","Data":"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"}
Sep 12 02:03:05.343051 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.343027    2582 scope.go:117] "RemoveContainer" containerID="0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"
Sep 12 02:03:05.344182 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.344153    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:05.344238 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.344224    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:05.344276 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:05.344256    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:06.356102 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:06.356066    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerStarted","Data":"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"}
Sep 12 02:03:06.427944 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:06.427909    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:06.428170 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:06.428145    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:06.428231 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:06.428209    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:07.428205 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:07.428170    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:07.428584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:07.428240    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:08.428251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:08.428213    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:08.428676 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:08.428268    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:09.428798 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:09.428763    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:09.429190 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:09.428818    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:10.427931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:10.427896    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:10.428119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:10.427953    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:11.428353 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:11.428303    2582 patch_prober.go:28] interesting pod/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 12 02:03:11.428854 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:11.428396    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 12 02:03:12.432033 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:12.432000    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:12.432399 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:12.432267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:12.432867 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:12.432850    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:24.839629 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:24.839594    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:03:24.840740 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:24.840469    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" podUID="f22e336c-6c51-4e73-a231-5bf284d1be75" containerName="maas-api" containerID="cri-o://d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf" gracePeriod=30
Sep 12 02:03:25.117670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.116444    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:03:25.256186 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca\") pod \"f22e336c-6c51-4e73-a231-5bf284d1be75\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") "
Sep 12 02:03:25.256384 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") pod \"f22e336c-6c51-4e73-a231-5bf284d1be75\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") "
Sep 12 02:03:25.256384 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs\") pod \"f22e336c-6c51-4e73-a231-5bf284d1be75\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") "
Sep 12 02:03:25.256384 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njnkj\" (UniqueName: \"kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj\") pod \"f22e336c-6c51-4e73-a231-5bf284d1be75\" (UID: \"f22e336c-6c51-4e73-a231-5bf284d1be75\") "
Sep 12 02:03:25.256580 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256482    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f22e336c-6c51-4e73-a231-5bf284d1be75" (UID: "f22e336c-6c51-4e73-a231-5bf284d1be75"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:03:25.256671 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.256638    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f22e336c-6c51-4e73-a231-5bf284d1be75-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:25.258576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.258536    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj" (OuterVolumeSpecName: "kube-api-access-njnkj") pod "f22e336c-6c51-4e73-a231-5bf284d1be75" (UID: "f22e336c-6c51-4e73-a231-5bf284d1be75"). InnerVolumeSpecName "kube-api-access-njnkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:03:25.258576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.258548    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f22e336c-6c51-4e73-a231-5bf284d1be75" (UID: "f22e336c-6c51-4e73-a231-5bf284d1be75"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:03:25.258757 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.258585    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f22e336c-6c51-4e73-a231-5bf284d1be75" (UID: "f22e336c-6c51-4e73-a231-5bf284d1be75"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:03:25.357528 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.357449    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:25.357528 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.357476    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22e336c-6c51-4e73-a231-5bf284d1be75-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:25.357528 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.357485    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnkj\" (UniqueName: \"kubernetes.io/projected/f22e336c-6c51-4e73-a231-5bf284d1be75-kube-api-access-njnkj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:25.459908 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.459870    2582 generic.go:356] "Generic (PLEG): container finished" podID="f22e336c-6c51-4e73-a231-5bf284d1be75" containerID="d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf" exitCode=0
Sep 12 02:03:25.460096 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.459912    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"
Sep 12 02:03:25.460096 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.459958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" event={"ID":"f22e336c-6c51-4e73-a231-5bf284d1be75","Type":"ContainerDied","Data":"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"}
Sep 12 02:03:25.460096 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.460009    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b" event={"ID":"f22e336c-6c51-4e73-a231-5bf284d1be75","Type":"ContainerDied","Data":"96487fe200be2cf9f1d5004b9b0e27265d3d63dd6ef657f8071d76eba81d7031"}
Sep 12 02:03:25.460096 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.460050    2582 scope.go:117] "RemoveContainer" containerID="d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"
Sep 12 02:03:25.469920 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.469900    2582 scope.go:117] "RemoveContainer" containerID="d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"
Sep 12 02:03:25.470222 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:03:25.470203    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf\": container with ID starting with d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf not found: ID does not exist" containerID="d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"
Sep 12 02:03:25.470321 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.470231    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf"} err="failed to get container status \"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf\": rpc error: code = NotFound desc = could not find container \"d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf\": container with ID starting with d3e1360150fb7e0d31c10310860c6a297e9e4b47e76619fedc3135057682bedf not found: ID does not exist"
Sep 12 02:03:25.483602 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.483578    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:03:25.488502 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:25.488479    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d5764ff8-7449488cf9-p8r5b"]
Sep 12 02:03:26.555816 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:26.555782    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22e336c-6c51-4e73-a231-5bf284d1be75" path="/var/lib/kubelet/pods/f22e336c-6c51-4e73-a231-5bf284d1be75/volumes"
Sep 12 02:03:32.038508 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.038476    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:03:32.038992 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.038694    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy" containerID="cri-o://795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253" gracePeriod=30
Sep 12 02:03:32.454104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.454071    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:03:32.454723 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.454696    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22e336c-6c51-4e73-a231-5bf284d1be75" containerName="maas-api"
Sep 12 02:03:32.454723 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.454724    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22e336c-6c51-4e73-a231-5bf284d1be75" containerName="maas-api"
Sep 12 02:03:32.454881 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.454842    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22e336c-6c51-4e73-a231-5bf284d1be75" containerName="maas-api"
Sep 12 02:03:32.458591 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.458568    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.461547 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.461520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-b99e21-openshift-default-dockercfg-r2q4b\""
Sep 12 02:03:32.473107 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.473081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:03:32.624574 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624574 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624580    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624651    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624695    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624748    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.624786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.624787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrb7\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725371    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725801 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrb7\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725853 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.725906 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.725860    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.726144 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.726111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.726278 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.726150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.726278 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.726169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.728068 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.728004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.728252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.728233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.741404 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.741384    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.741512 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.741492    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrb7\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7\") pod \"e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") " pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:32.769515 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:32.769490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:33.045090 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.045004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:03:33.051877 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.051830    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:33.052008 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.051965    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:33.052008 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.052005    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:03:33.503016 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.502972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" event={"ID":"a7033ee5-fd35-488e-9b1e-77ce13ca614f","Type":"ContainerStarted","Data":"2cea2bc3cc9a3699b896692f0849b98b81ddc2d0991a55fa339080403983ce1e"}
Sep 12 02:03:33.503218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.503101    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" event={"ID":"a7033ee5-fd35-488e-9b1e-77ce13ca614f","Type":"ContainerStarted","Data":"aaaeecdd5465974ee6397d83b66a1315d2f29b541f6e7ca5ed05dc5b40b42d9b"}
Sep 12 02:03:33.527429 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.527369    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" podStartSLOduration=1.527353644 podStartE2EDuration="1.527353644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:03:33.52450683 +0000 UTC m=+1107.451452945" watchObservedRunningTime="2026-09-12 02:03:33.527353644 +0000 UTC m=+1107.454299530"
Sep 12 02:03:33.770002 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:33.769926    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:34.775647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:34.775613    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:35.514345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:35.514308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:35.515301 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:35.515278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:03:37.298191 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.298167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:37.371631 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371611    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371718    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371758    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371777    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.371802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371802    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.372147 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9x8c\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.372147 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371901    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:03:37.372147 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.371929    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data\") pod \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\" (UID: \"d97ce6e1-0661-4fed-9ce1-8ffe6e940379\") "
Sep 12 02:03:37.372147 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372104    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:03:37.372321 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:03:37.372321 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372299    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data" (OuterVolumeSpecName: "istio-data") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:03:37.372419 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372361    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.372419 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372380    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.372419 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372394    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.372419 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.372405    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:03:37.374006 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.373980    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:03:37.374138 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.374128    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token" (OuterVolumeSpecName: "istio-token") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:03:37.374289 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.374264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c" (OuterVolumeSpecName: "kube-api-access-q9x8c") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "kube-api-access-q9x8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:03:37.374459 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.374439    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d97ce6e1-0661-4fed-9ce1-8ffe6e940379" (UID: "d97ce6e1-0661-4fed-9ce1-8ffe6e940379"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:03:37.473742 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473714    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.473742 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473737    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.473742 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473746    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.473931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473754    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.473931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473763    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9x8c\" (UniqueName: \"kubernetes.io/projected/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-kube-api-access-q9x8c\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.473931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.473770    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d97ce6e1-0661-4fed-9ce1-8ffe6e940379-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:03:37.525792 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.525766    2582 generic.go:356] "Generic (PLEG): container finished" podID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerID="795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253" exitCode=0
Sep 12 02:03:37.525924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.525826    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"
Sep 12 02:03:37.525924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.525854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerDied","Data":"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"}
Sep 12 02:03:37.525924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.525904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w" event={"ID":"d97ce6e1-0661-4fed-9ce1-8ffe6e940379","Type":"ContainerDied","Data":"d4f52eabe21806e5840042f722f7c94908a66f3c8fed77e0afa1f33c5f89cb3c"}
Sep 12 02:03:37.526076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.525928    2582 scope.go:117] "RemoveContainer" containerID="795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"
Sep 12 02:03:37.540557 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.539553    2582 scope.go:117] "RemoveContainer" containerID="0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"
Sep 12 02:03:37.548300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.548280    2582 scope.go:117] "RemoveContainer" containerID="795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"
Sep 12 02:03:37.548577 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:03:37.548557    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253\": container with ID starting with 795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253 not found: ID does not exist" containerID="795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"
Sep 12 02:03:37.548647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.548583    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253"} err="failed to get container status \"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253\": rpc error: code = NotFound desc = could not find container \"795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253\": container with ID starting with 795cadd6ffee804aa3ee11d36c6f117cbe56ad60e42dceb88785f69feefee253 not found: ID does not exist"
Sep 12 02:03:37.548647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.548601    2582 scope.go:117] "RemoveContainer" containerID="0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"
Sep 12 02:03:37.548843 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:03:37.548824    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880\": container with ID starting with 0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880 not found: ID does not exist" containerID="0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"
Sep 12 02:03:37.548883 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.548849    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880"} err="failed to get container status \"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880\": rpc error: code = NotFound desc = could not find container \"0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880\": container with ID starting with 0a10e02fc2d3fa706b06bad4507ea85e69a144adec6be36f614371bfa7d75880 not found: ID does not exist"
Sep 12 02:03:37.552970 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.552949    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:03:37.556388 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:37.556367    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d5764ff8-openshift-default-857b87b644-dgk5w"]
Sep 12 02:03:38.555416 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:38.555387    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" path="/var/lib/kubelet/pods/d97ce6e1-0661-4fed-9ce1-8ffe6e940379/volumes"
Sep 12 02:03:52.603599 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.603558    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:03:52.603986 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.603970    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:03:52.604032 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.603987    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:03:52.604093 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.604078    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:03:52.604093 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.604087    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:03:52.608524 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.608498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.611337 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.611313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-e8046f\""
Sep 12 02:03:52.611511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.611490    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-e8046f-dockercfg-qqkvs\""
Sep 12 02:03:52.618848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.618825    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:03:52.714406 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.714364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchkj\" (UniqueName: \"kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.714597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.714520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.815424 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.815382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchkj\" (UniqueName: \"kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.815615 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.815444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.816063 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.816025    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.823165 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.823142    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchkj\" (UniqueName: \"kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj\") pod \"payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:52.920733 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:52.920703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:53.191952 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:53.191862    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:03:53.194806 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:03:53.194780    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736c0762_6a3a_45fd_941a_aa2c0cd6d007.slice/crio-be2d470c7a9a35861b0443523f0fe3bf7cfd27640b7dc0e10a60af41b559d75e WatchSource:0}: Error finding container be2d470c7a9a35861b0443523f0fe3bf7cfd27640b7dc0e10a60af41b559d75e: Status 404 returned error can't find the container with id be2d470c7a9a35861b0443523f0fe3bf7cfd27640b7dc0e10a60af41b559d75e
Sep 12 02:03:53.611768 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:53.611691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" event={"ID":"736c0762-6a3a-45fd-941a-aa2c0cd6d007","Type":"ContainerStarted","Data":"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"}
Sep 12 02:03:53.611768 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:53.611739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" event={"ID":"736c0762-6a3a-45fd-941a-aa2c0cd6d007","Type":"ContainerStarted","Data":"be2d470c7a9a35861b0443523f0fe3bf7cfd27640b7dc0e10a60af41b559d75e"}
Sep 12 02:03:53.612185 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:53.611779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:03:53.629122 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:03:53.629069    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" podStartSLOduration=1.629018867 podStartE2EDuration="1.629018867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:03:53.628109276 +0000 UTC m=+1127.555055163" watchObservedRunningTime="2026-09-12 02:03:53.629018867 +0000 UTC m=+1127.555964753"
Sep 12 02:04:04.617934 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:04.617902    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:04:28.214331 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.214296    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:04:28.214762 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.214689    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:04:28.214762 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.214699    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d97ce6e1-0661-4fed-9ce1-8ffe6e940379" containerName="istio-proxy"
Sep 12 02:04:28.218117 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.218099    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.220796 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.220771    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"kube-root-ca.crt\""
Sep 12 02:04:28.221807 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.221775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"default-dockercfg-494kz\""
Sep 12 02:04:28.221923 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.221779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"e2e-conflict-b-3901f64e-kserve-self-signed-certs\""
Sep 12 02:04:28.221923 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.221779    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-98f68e\"/\"openshift-service-ca.crt\""
Sep 12 02:04:28.226822 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.226800    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:04:28.244833 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.244811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.244931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.244844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.244931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.244875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.245011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.244996    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.245072 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.245023    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.245113 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.245094    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5hw2\" (UniqueName: \"kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346393 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5hw2\" (UniqueName: \"kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346493 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346534 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346595 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346724 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346724 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346866 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346845    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.346973 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.346937    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.348852 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.348829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.349119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.349102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.356661 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.356643    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5hw2\" (UniqueName: \"kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2\") pod \"e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") " pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.527826 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.527774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:28.801226 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:28.801133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:04:28.807114 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:04:28.807076    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd91ac3e7_d6b7_4ec4_aa68_dae9378f7789.slice/crio-47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f WatchSource:0}: Error finding container 47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f: Status 404 returned error can't find the container with id 47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f
Sep 12 02:04:29.788330 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:29.788292    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" event={"ID":"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789","Type":"ContainerStarted","Data":"8aeb6d21dd02bc90a0a0d76dfe143cac7cb5a509eeaa81825f503c5691866c81"}
Sep 12 02:04:29.788712 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:29.788340    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" event={"ID":"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789","Type":"ContainerStarted","Data":"47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f"}
Sep 12 02:04:29.805930 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:29.805880    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" podStartSLOduration=1.805865372 podStartE2EDuration="1.805865372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:04:29.804383722 +0000 UTC m=+1163.731329608" watchObservedRunningTime="2026-09-12 02:04:29.805865372 +0000 UTC m=+1163.732811309"
Sep 12 02:04:38.528559 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:38.528524    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:38.529032 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:38.528604    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:38.541347 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:38.541325    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:38.843500 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:38.843413    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:04:46.757836 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:46.757806    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:04:46.758254 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:46.758082    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerName="istio-proxy" containerID="cri-o://2cea2bc3cc9a3699b896692f0849b98b81ddc2d0991a55fa339080403983ce1e" gracePeriod=30
Sep 12 02:04:50.517116 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:50.517077    2582 patch_prober.go:28] interesting pod/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:04:50.517503 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:50.517145    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:04:50.620457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:50.620405    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"]
Sep 12 02:04:50.622616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:50.622590    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-3fc028-29819640-l46sm"]
Sep 12 02:04:51.903679 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:51.903645    2582 generic.go:356] "Generic (PLEG): container finished" podID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerID="2cea2bc3cc9a3699b896692f0849b98b81ddc2d0991a55fa339080403983ce1e" exitCode=0
Sep 12 02:04:51.904077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:51.903717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" event={"ID":"a7033ee5-fd35-488e-9b1e-77ce13ca614f","Type":"ContainerDied","Data":"2cea2bc3cc9a3699b896692f0849b98b81ddc2d0991a55fa339080403983ce1e"}
Sep 12 02:04:52.021318 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.021259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:04:52.175556 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175518    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.175748 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.175748 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.175748 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175656    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.175923 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175859    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlrb7\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.176005 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.176098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:04:52.176098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.176098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.175910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:04:52.176098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176004    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:04:52.176333 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176108    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.176333 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176135    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token\") pod \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\" (UID: \"a7033ee5-fd35-488e-9b1e-77ce13ca614f\") "
Sep 12 02:04:52.176333 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176258    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:04:52.176492 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:04:52.176577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176555    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.176637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176583    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.176637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176599    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.176637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176611    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.176637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.176625    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.177978 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.177953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:04:52.178121 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.178089    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7" (OuterVolumeSpecName: "kube-api-access-jlrb7") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "kube-api-access-jlrb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:04:52.178201 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.178176    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:04:52.178439 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.178410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7033ee5-fd35-488e-9b1e-77ce13ca614f" (UID: "a7033ee5-fd35-488e-9b1e-77ce13ca614f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:04:52.277624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.277532    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.277624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.277561    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.277624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.277571    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7033ee5-fd35-488e-9b1e-77ce13ca614f-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.277624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.277579    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlrb7\" (UniqueName: \"kubernetes.io/projected/a7033ee5-fd35-488e-9b1e-77ce13ca614f-kube-api-access-jlrb7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:04:52.556201 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.556122    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8d24fa-8c7f-4dd0-b165-65acd2641b89" path="/var/lib/kubelet/pods/5a8d24fa-8c7f-4dd0-b165-65acd2641b89/volumes"
Sep 12 02:04:52.909990 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.909960    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62" event={"ID":"a7033ee5-fd35-488e-9b1e-77ce13ca614f","Type":"ContainerDied","Data":"aaaeecdd5465974ee6397d83b66a1315d2f29b541f6e7ca5ed05dc5b40b42d9b"}
Sep 12 02:04:52.909990 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.909990    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"
Sep 12 02:04:52.910504 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.910030    2582 scope.go:117] "RemoveContainer" containerID="2cea2bc3cc9a3699b896692f0849b98b81ddc2d0991a55fa339080403983ce1e"
Sep 12 02:04:52.936783 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.936756    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:04:52.941234 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:52.941203    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-b99e21-openshift-default-7bc4479c7b-xds62"]
Sep 12 02:04:54.555779 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:54.555740    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" path="/var/lib/kubelet/pods/a7033ee5-fd35-488e-9b1e-77ce13ca614f/volumes"
Sep 12 02:04:57.224323 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:57.224292    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 02:04:57.224815 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:04:57.224551    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" containerID="cri-o://f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556" gracePeriod=30
Sep 12 02:05:00.520772 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.520707    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:05:00.521399 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.521375    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerName="istio-proxy"
Sep 12 02:05:00.521460 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.521404    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerName="istio-proxy"
Sep 12 02:05:00.521536 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.521524    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7033ee5-fd35-488e-9b1e-77ce13ca614f" containerName="istio-proxy"
Sep 12 02:05:00.526584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.526563    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.529424 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.529399    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-60c2473b-openshift-default-dockercfg-hsnxz\""
Sep 12 02:05:00.537273 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.537243    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:05:00.558868 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.558843    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.558885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.558938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.558975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.559015    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.559033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.559097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62crc\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.559218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.559356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.559282    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660246 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660204    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660246 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660250    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660294    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660344    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62crc\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660511 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660458    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660835 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.660835 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.661017 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.660991    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.661130 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.661052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.661130 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.661059    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.661130 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.661103    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.662751 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.662720    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.662940 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.662925    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.671551 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.671524    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.671846 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.671826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62crc\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc\") pod \"e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") " pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:00.841734 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:00.841394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:01.124978 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.124954    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:05:01.126957 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:05:01.126930    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ffb94e_42e4_479e_8dc2_044fdd065761.slice/crio-a0eebd95dc51a4e46987d2cc76f8d69c7fd862bc78363036813f25432f5107e7 WatchSource:0}: Error finding container a0eebd95dc51a4e46987d2cc76f8d69c7fd862bc78363036813f25432f5107e7: Status 404 returned error can't find the container with id a0eebd95dc51a4e46987d2cc76f8d69c7fd862bc78363036813f25432f5107e7
Sep 12 02:05:01.129157 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.129121    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:01.129265 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.129188    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:01.129265 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.129221    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:01.829485 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.829453    2582 patch_prober.go:28] interesting pod/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:05:01.829848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.829502    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:05:01.962579 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.962545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerStarted","Data":"0faaf1eb01e17e74f78aec69b371bdc3f3450c66844d5a1365b6888b038c825a"}
Sep 12 02:05:01.962740 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.962598    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerStarted","Data":"a0eebd95dc51a4e46987d2cc76f8d69c7fd862bc78363036813f25432f5107e7"}
Sep 12 02:05:01.983417 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:01.983372    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podStartSLOduration=1.983358078 podStartE2EDuration="1.983358078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:01.981828834 +0000 UTC m=+1195.908774722" watchObservedRunningTime="2026-09-12 02:05:01.983358078 +0000 UTC m=+1195.910303964"
Sep 12 02:05:02.477700 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.477676    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 02:05:02.581253 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581181    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581253 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581223    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581285    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581334    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581356    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85xkn\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581408    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581463 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581434    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs\") pod \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\" (UID: \"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3\") "
Sep 12 02:05:02.581816 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581582    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:05:02.581816 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:05:02.581816 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:05:02.581975 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581830    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:05:02.581975 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581920    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.581975 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581943    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.581975 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581959    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.582183 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.581973    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.582183 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.582091    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:05:02.583646 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.583617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:05:02.583646 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.583632    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn" (OuterVolumeSpecName: "kube-api-access-85xkn") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "kube-api-access-85xkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:05:02.583804 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.583750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:05:02.583845 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.583823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" (UID: "1a28a19b-9b97-4110-8bb6-a5125ae1bcc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:05:02.682894 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.682863    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.682894 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.682890    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.682894 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.682899    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85xkn\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-kube-api-access-85xkn\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.683125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.682908    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.683125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.682916    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:02.843154 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.843076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:02.848508 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.848477    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:02.969128 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969094    2582 generic.go:356] "Generic (PLEG): container finished" podID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerID="f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556" exitCode=0
Sep 12 02:05:02.969279 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969164    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"
Sep 12 02:05:02.969279 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969174    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerDied","Data":"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"}
Sep 12 02:05:02.969279 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld" event={"ID":"1a28a19b-9b97-4110-8bb6-a5125ae1bcc3","Type":"ContainerDied","Data":"160e498437b736c337c876e4e805da3d68a0bf46100342f12e163cfab552ac41"}
Sep 12 02:05:02.969279 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969244    2582 scope.go:117] "RemoveContainer" containerID="f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"
Sep 12 02:05:02.969595 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.969575    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:02.970484 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.970474    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:02.979494 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.979479    2582 scope.go:117] "RemoveContainer" containerID="74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"
Sep 12 02:05:02.987814 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.987796    2582 scope.go:117] "RemoveContainer" containerID="f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"
Sep 12 02:05:02.988280 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:05:02.988249    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556\": container with ID starting with f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556 not found: ID does not exist" containerID="f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"
Sep 12 02:05:02.988379 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.988288    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556"} err="failed to get container status \"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556\": rpc error: code = NotFound desc = could not find container \"f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556\": container with ID starting with f7762ec3bfdb858ef89fa540e8388331745b47b569c24e6f36b5d7f2ec154556 not found: ID does not exist"
Sep 12 02:05:02.988379 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.988313    2582 scope.go:117] "RemoveContainer" containerID="74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"
Sep 12 02:05:02.988642 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:05:02.988622    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4\": container with ID starting with 74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4 not found: ID does not exist" containerID="74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"
Sep 12 02:05:02.988715 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:02.988647    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4"} err="failed to get container status \"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4\": rpc error: code = NotFound desc = could not find container \"74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4\": container with ID starting with 74e9d2c3acc035abdde028ec18f86095b3395337070aa3932e286f34c65f45b4 not found: ID does not exist"
Sep 12 02:05:03.011414 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:03.011385    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 02:05:03.014281 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:03.014257    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3fc028-openshift-default-8448b86b78-7ftld"]
Sep 12 02:05:04.556497 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:04.556458    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" path="/var/lib/kubelet/pods/1a28a19b-9b97-4110-8bb6-a5125ae1bcc3/volumes"
Sep 12 02:05:06.589851 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:06.589823    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:05:06.590560 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:06.589895    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:05:08.682046 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.682001    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:05:08.682451 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.682437    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:08.682451 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.682452    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:08.682542 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.682533    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:08.682575 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.682544    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:08.687000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.686984    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.689306 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.689281    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-60c2473b\""
Sep 12 02:05:08.689432 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.689285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-60c2473b-dockercfg-jdgsb\""
Sep 12 02:05:08.695886 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.695867    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:05:08.739483 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.739223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.739483 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.739297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7v7\" (UniqueName: \"kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.840360 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.840330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.840528 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.840384    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7v7\" (UniqueName: \"kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.841240 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.841221    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.848914 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.848890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7v7\" (UniqueName: \"kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7\") pod \"payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") " pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:08.997533 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:08.997456    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:09.275663 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:09.275638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:05:09.277584 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:05:09.277555    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97df382_5559_4845_bff4_ad2da1e39431.slice/crio-79eb68f7000dcf50e54ca0ee2e3be6e8747979f9c63eccdc6ea8379257edb949 WatchSource:0}: Error finding container 79eb68f7000dcf50e54ca0ee2e3be6e8747979f9c63eccdc6ea8379257edb949: Status 404 returned error can't find the container with id 79eb68f7000dcf50e54ca0ee2e3be6e8747979f9c63eccdc6ea8379257edb949
Sep 12 02:05:10.011601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.011564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" event={"ID":"f97df382-5559-4845-bff4-ad2da1e39431","Type":"ContainerStarted","Data":"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"}
Sep 12 02:05:10.011980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.011632    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" event={"ID":"f97df382-5559-4845-bff4-ad2da1e39431","Type":"ContainerStarted","Data":"79eb68f7000dcf50e54ca0ee2e3be6e8747979f9c63eccdc6ea8379257edb949"}
Sep 12 02:05:10.011980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.011754    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:10.028121 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.028070    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" podStartSLOduration=2.02805397 podStartE2EDuration="2.02805397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:10.026972151 +0000 UTC m=+1203.953918039" watchObservedRunningTime="2026-09-12 02:05:10.02805397 +0000 UTC m=+1203.954999847"
Sep 12 02:05:10.264110 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.264017    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 02:05:10.264384 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.264335    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" podUID="31de3ece-f720-42b9-aeca-e6a306e1b249" containerName="maas-api" containerID="cri-o://e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c" gracePeriod=30
Sep 12 02:05:10.318172 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.318140    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 02:05:10.318397 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.318370    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" podUID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" containerName="payload-processing" containerID="cri-o://2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4" gracePeriod=30
Sep 12 02:05:10.560691 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.560665    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 02:05:10.603569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.603542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 02:05:10.660293 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srzm6\" (UniqueName: \"kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6\") pod \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") "
Sep 12 02:05:10.660470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca\") pod \"31de3ece-f720-42b9-aeca-e6a306e1b249\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") "
Sep 12 02:05:10.660470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660330    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume\") pod \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\" (UID: \"ad517be6-9b16-44eb-a5a9-2a1b8283282f\") "
Sep 12 02:05:10.660470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660372    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs\") pod \"31de3ece-f720-42b9-aeca-e6a306e1b249\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") "
Sep 12 02:05:10.660470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660404    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls\") pod \"31de3ece-f720-42b9-aeca-e6a306e1b249\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") "
Sep 12 02:05:10.660470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrps6\" (UniqueName: \"kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6\") pod \"31de3ece-f720-42b9-aeca-e6a306e1b249\" (UID: \"31de3ece-f720-42b9-aeca-e6a306e1b249\") "
Sep 12 02:05:10.660736 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660697    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "31de3ece-f720-42b9-aeca-e6a306e1b249" (UID: "31de3ece-f720-42b9-aeca-e6a306e1b249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:05:10.660791 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660732    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad517be6-9b16-44eb-a5a9-2a1b8283282f" (UID: "ad517be6-9b16-44eb-a5a9-2a1b8283282f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:05:10.660956 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660936    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31de3ece-f720-42b9-aeca-e6a306e1b249-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:10.661068 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.660961    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad517be6-9b16-44eb-a5a9-2a1b8283282f-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:10.662544 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.662512    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "31de3ece-f720-42b9-aeca-e6a306e1b249" (UID: "31de3ece-f720-42b9-aeca-e6a306e1b249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:05:10.662647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.662624    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6" (OuterVolumeSpecName: "kube-api-access-srzm6") pod "ad517be6-9b16-44eb-a5a9-2a1b8283282f" (UID: "ad517be6-9b16-44eb-a5a9-2a1b8283282f"). InnerVolumeSpecName "kube-api-access-srzm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:05:10.662924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.662904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "31de3ece-f720-42b9-aeca-e6a306e1b249" (UID: "31de3ece-f720-42b9-aeca-e6a306e1b249"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:05:10.662924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.662910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6" (OuterVolumeSpecName: "kube-api-access-hrps6") pod "31de3ece-f720-42b9-aeca-e6a306e1b249" (UID: "31de3ece-f720-42b9-aeca-e6a306e1b249"). InnerVolumeSpecName "kube-api-access-hrps6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:05:10.761576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.761538    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzm6\" (UniqueName: \"kubernetes.io/projected/ad517be6-9b16-44eb-a5a9-2a1b8283282f-kube-api-access-srzm6\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:10.761576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.761571    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:10.761576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.761580    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31de3ece-f720-42b9-aeca-e6a306e1b249-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:10.761806 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:10.761588    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrps6\" (UniqueName: \"kubernetes.io/projected/31de3ece-f720-42b9-aeca-e6a306e1b249-kube-api-access-hrps6\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:05:11.018107 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.018066    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" containerID="2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4" exitCode=0
Sep 12 02:05:11.018547 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.018121    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"
Sep 12 02:05:11.018547 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.018159    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" event={"ID":"ad517be6-9b16-44eb-a5a9-2a1b8283282f","Type":"ContainerDied","Data":"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"}
Sep 12 02:05:11.018547 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.018260    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9" event={"ID":"ad517be6-9b16-44eb-a5a9-2a1b8283282f","Type":"ContainerDied","Data":"a324f77843c17f24290b9592b3c82e11ccf5c542314dcdf153790b7e14cbc515"}
Sep 12 02:05:11.018547 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.018292    2582 scope.go:117] "RemoveContainer" containerID="2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"
Sep 12 02:05:11.019540 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.019517    2582 generic.go:356] "Generic (PLEG): container finished" podID="31de3ece-f720-42b9-aeca-e6a306e1b249" containerID="e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c" exitCode=0
Sep 12 02:05:11.019616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.019586    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"
Sep 12 02:05:11.019616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.019601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" event={"ID":"31de3ece-f720-42b9-aeca-e6a306e1b249","Type":"ContainerDied","Data":"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"}
Sep 12 02:05:11.019697 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.019641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5" event={"ID":"31de3ece-f720-42b9-aeca-e6a306e1b249","Type":"ContainerDied","Data":"0efd0abd4c79f45a1d6cc0aea74f7c1ecae6ec6eed030efcf60fd7d9dcdfa6aa"}
Sep 12 02:05:11.027401 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.027380    2582 scope.go:117] "RemoveContainer" containerID="2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"
Sep 12 02:05:11.027697 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:05:11.027671    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4\": container with ID starting with 2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4 not found: ID does not exist" containerID="2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"
Sep 12 02:05:11.027782 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.027707    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4"} err="failed to get container status \"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4\": rpc error: code = NotFound desc = could not find container \"2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4\": container with ID starting with 2e5f840869e0ac16e714d6b63f55114fbc1f15d9fa26bad120dc55e8ec53e2b4 not found: ID does not exist"
Sep 12 02:05:11.027782 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.027731    2582 scope.go:117] "RemoveContainer" containerID="e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"
Sep 12 02:05:11.035903 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.035888    2582 scope.go:117] "RemoveContainer" containerID="e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"
Sep 12 02:05:11.036186 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:05:11.036165    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c\": container with ID starting with e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c not found: ID does not exist" containerID="e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"
Sep 12 02:05:11.036246 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.036192    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c"} err="failed to get container status \"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c\": rpc error: code = NotFound desc = could not find container \"e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c\": container with ID starting with e4e19c946207f8f6f33094bafc646d63334f2cd6116391b382ecd1676f77c84c not found: ID does not exist"
Sep 12 02:05:11.044363 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.044344    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 02:05:11.047465 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.047439    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7d6e9f-c6474fc8f-7czp9"]
Sep 12 02:05:11.061466 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.061447    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 02:05:11.065172 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:11.065152    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7d6e9f-77d4988c9c-6pqn5"]
Sep 12 02:05:12.554890 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:12.554859    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31de3ece-f720-42b9-aeca-e6a306e1b249" path="/var/lib/kubelet/pods/31de3ece-f720-42b9-aeca-e6a306e1b249/volumes"
Sep 12 02:05:12.555267 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:12.555237    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" path="/var/lib/kubelet/pods/ad517be6-9b16-44eb-a5a9-2a1b8283282f/volumes"
Sep 12 02:05:21.022419 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:21.022387    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:05:30.128930 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.128896    2582 generic.go:356] "Generic (PLEG): container finished" podID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerID="0faaf1eb01e17e74f78aec69b371bdc3f3450c66844d5a1365b6888b038c825a" exitCode=0
Sep 12 02:05:30.129471 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.128954    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerDied","Data":"0faaf1eb01e17e74f78aec69b371bdc3f3450c66844d5a1365b6888b038c825a"}
Sep 12 02:05:30.129546 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.129472    2582 scope.go:117] "RemoveContainer" containerID="0faaf1eb01e17e74f78aec69b371bdc3f3450c66844d5a1365b6888b038c825a"
Sep 12 02:05:30.130673 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.130639    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:30.130765 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.130733    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:30.130803 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:30.130768    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:31.135910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.135876    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerStarted","Data":"fb84daa7586ed6108cde0e4af39e288ad5f64e36d8a7e2765c05862ee2048bc7"}
Sep 12 02:05:31.348988 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.348954    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:05:31.349376 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349362    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" containerName="payload-processing"
Sep 12 02:05:31.349427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349378    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" containerName="payload-processing"
Sep 12 02:05:31.349427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349389    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31de3ece-f720-42b9-aeca-e6a306e1b249" containerName="maas-api"
Sep 12 02:05:31.349427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349396    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="31de3ece-f720-42b9-aeca-e6a306e1b249" containerName="maas-api"
Sep 12 02:05:31.349427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349405    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:31.349427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349411    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a28a19b-9b97-4110-8bb6-a5125ae1bcc3" containerName="istio-proxy"
Sep 12 02:05:31.349596 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349473    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="31de3ece-f720-42b9-aeca-e6a306e1b249" containerName="maas-api"
Sep 12 02:05:31.349596 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.349482    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad517be6-9b16-44eb-a5a9-2a1b8283282f" containerName="payload-processing"
Sep 12 02:05:31.352903 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.352883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.356180 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.355968    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-a744d5\""
Sep 12 02:05:31.356180 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.356006    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-a744d5-dockercfg-xbd6d\""
Sep 12 02:05:31.367942 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.367914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:05:31.458543 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.458455    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbww7\" (UniqueName: \"kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.458714 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.458545    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.559353 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.559317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbww7\" (UniqueName: \"kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.559534 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.559375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.560001 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.559980    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.567276 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.567245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbww7\" (UniqueName: \"kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7\") pod \"payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.670210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.670170    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:31.842221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.842184    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:31.843326 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.843300    2582 patch_prober.go:28] interesting pod/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 12 02:05:31.843472 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.843356    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 12 02:05:31.957517 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:31.957481    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:05:31.960704 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:05:31.960678    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf81eea7_29b0_48c0_a027_abdffc079c18.slice/crio-cd78c612fc8c002a7284f467bd8f2ddc69c2f02b9d7a5b28daa8c785fb782f05 WatchSource:0}: Error finding container cd78c612fc8c002a7284f467bd8f2ddc69c2f02b9d7a5b28daa8c785fb782f05: Status 404 returned error can't find the container with id cd78c612fc8c002a7284f467bd8f2ddc69c2f02b9d7a5b28daa8c785fb782f05
Sep 12 02:05:32.142849 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.142811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" event={"ID":"bf81eea7-29b0-48c0-a027-abdffc079c18","Type":"ContainerStarted","Data":"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"}
Sep 12 02:05:32.143252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.142872    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" event={"ID":"bf81eea7-29b0-48c0-a027-abdffc079c18","Type":"ContainerStarted","Data":"cd78c612fc8c002a7284f467bd8f2ddc69c2f02b9d7a5b28daa8c785fb782f05"}
Sep 12 02:05:32.143252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.143075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:32.160026 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.159965    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" podStartSLOduration=1.159946798 podStartE2EDuration="1.159946798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:32.157270382 +0000 UTC m=+1226.084216268" watchObservedRunningTime="2026-09-12 02:05:32.159946798 +0000 UTC m=+1226.086892686"
Sep 12 02:05:32.842749 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.842708    2582 patch_prober.go:28] interesting pod/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 12 02:05:32.842957 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.842770    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 12 02:05:32.969580 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:32.969551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:33.842856 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:33.842819    2582 patch_prober.go:28] interesting pod/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 12 02:05:33.843261 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:33.842878    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 12 02:05:34.842367 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:34.842335    2582 patch_prober.go:28] interesting pod/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 12 02:05:34.842554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:34.842396    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 12 02:05:35.843077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:35.843021    2582 patch_prober.go:28] interesting pod/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 12 02:05:35.843457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:35.843097    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 12 02:05:36.847329 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:36.847294    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:36.848652 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:36.848627    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:05:43.149413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:43.149377    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:05:57.292105 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.292073    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:05:57.295946 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.295922    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.300375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.300321    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-fbb6d9-openshift-default-dockercfg-8t8sh\""
Sep 12 02:05:57.309356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.309334    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:05:57.405630 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.405817 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.405817 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.405817 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5d84\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.405817 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.406082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.406082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.406082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.406082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.405902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.506826 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.506788    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5d84\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507002 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.506856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507002 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.506929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507002 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.506978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507174    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507207    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507448 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507246    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507448 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507525 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507446    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507596 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507633 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.507935 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.507911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.509900 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.509872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.510004 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.509948    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.514847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.514821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.515162 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.515146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5d84\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84\") pod \"e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") " pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.607146 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.607058    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:57.888245 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.888217    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:05:57.889671 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:05:57.889643    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76dfbc5_3a08_4bfb_81dd_91115122220c.slice/crio-b085b24f69b7fac64756dd50ddbed06f823ad348b28beb22512a9e97aef4a29c WatchSource:0}: Error finding container b085b24f69b7fac64756dd50ddbed06f823ad348b28beb22512a9e97aef4a29c: Status 404 returned error can't find the container with id b085b24f69b7fac64756dd50ddbed06f823ad348b28beb22512a9e97aef4a29c
Sep 12 02:05:57.891972 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.891934    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:57.892100 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.892015    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:57.892100 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:57.892075    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:05:58.282654 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:58.282567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerStarted","Data":"54e860885570ee9dd8e10da9a54c2ea22608bcb9b494c13dc83bef92424140ef"}
Sep 12 02:05:58.282654 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:58.282635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerStarted","Data":"b085b24f69b7fac64756dd50ddbed06f823ad348b28beb22512a9e97aef4a29c"}
Sep 12 02:05:58.304862 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:58.304812    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podStartSLOduration=1.304796885 podStartE2EDuration="1.304796885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:05:58.303899578 +0000 UTC m=+1252.230845464" watchObservedRunningTime="2026-09-12 02:05:58.304796885 +0000 UTC m=+1252.231742824"
Sep 12 02:05:58.607985 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:58.607904    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:58.613669 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:58.613638    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:59.103349 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:59.103317    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:05:59.288199 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:59.288167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:05:59.289238 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:05:59.289213    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:06:05.071556 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.071522    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:06:05.075420 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.075397    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.078047 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.077995    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-fbb6d9\""
Sep 12 02:06:05.078171 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.078107    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-fbb6d9-dockercfg-cslvm\""
Sep 12 02:06:05.089093 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.089067    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:06:05.182699 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.182655    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.182984 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.182961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwts\" (UniqueName: \"kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.284654 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.284614    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.284812 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.284672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwts\" (UniqueName: \"kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.285366 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.285348    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.292746 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.292717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwts\" (UniqueName: \"kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts\") pod \"payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.387823 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.387795    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:05.660100 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:05.660071    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:06:05.661979 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:06:05.661953    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f5dff4_ba23_4665_b056_254fc7ead5df.slice/crio-4795d48dbf4237711eb41008a79d06ee88b6e638e898b9d5f144253a250735b9 WatchSource:0}: Error finding container 4795d48dbf4237711eb41008a79d06ee88b6e638e898b9d5f144253a250735b9: Status 404 returned error can't find the container with id 4795d48dbf4237711eb41008a79d06ee88b6e638e898b9d5f144253a250735b9
Sep 12 02:06:06.333015 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:06.332971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" event={"ID":"45f5dff4-ba23-4665-b056-254fc7ead5df","Type":"ContainerStarted","Data":"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"}
Sep 12 02:06:06.333584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:06.333171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:06.333584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:06.333215    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" event={"ID":"45f5dff4-ba23-4665-b056-254fc7ead5df","Type":"ContainerStarted","Data":"4795d48dbf4237711eb41008a79d06ee88b6e638e898b9d5f144253a250735b9"}
Sep 12 02:06:06.349024 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:06.348976    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" podStartSLOduration=1.348963333 podStartE2EDuration="1.348963333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:06:06.347829135 +0000 UTC m=+1260.274775021" watchObservedRunningTime="2026-09-12 02:06:06.348963333 +0000 UTC m=+1260.275909255"
Sep 12 02:06:06.638281 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:06.638248    2582 scope.go:117] "RemoveContainer" containerID="eff1ff2f989e84367f38cd4ae74dbc8621a0e8d16a9bdb320eb1093f1856b9ac"
Sep 12 02:06:14.124249 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.124195    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" podUID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" containerName="main" containerID="cri-o://8aeb6d21dd02bc90a0a0d76dfe143cac7cb5a509eeaa81825f503c5691866c81" gracePeriod=45
Sep 12 02:06:14.374407 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.374345    2582 generic.go:356] "Generic (PLEG): container finished" podID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" containerID="8aeb6d21dd02bc90a0a0d76dfe143cac7cb5a509eeaa81825f503c5691866c81" exitCode=0
Sep 12 02:06:14.374506 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.374411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" event={"ID":"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789","Type":"ContainerDied","Data":"8aeb6d21dd02bc90a0a0d76dfe143cac7cb5a509eeaa81825f503c5691866c81"}
Sep 12 02:06:14.374506 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.374462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl" event={"ID":"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789","Type":"ContainerDied","Data":"47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f"}
Sep 12 02:06:14.374506 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.374475    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47eb05d12e18b46023b95513b45dd0541ddef87dd2eef272744bbdf65d72451f"
Sep 12 02:06:14.382745 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.382726    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:06:14.474917 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.474892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475074 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475074 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475061    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475149 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475099    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5hw2\" (UniqueName: \"kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475149 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475255 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475185    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs\") pod \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\" (UID: \"d91ac3e7-d6b7-4ec4-aa68-dae9378f7789\") "
Sep 12 02:06:14.475255 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475200    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home" (OuterVolumeSpecName: "home") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:06:14.475356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:06:14.475356 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475304    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache" (OuterVolumeSpecName: "model-cache") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:06:14.475500 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475480    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:14.475581 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475512    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:14.475581 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.475528    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:14.477359 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.477336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm" (OuterVolumeSpecName: "dshm") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:06:14.477451 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.477385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2" (OuterVolumeSpecName: "kube-api-access-q5hw2") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "kube-api-access-q5hw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:06:14.477451 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.477421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" (UID: "d91ac3e7-d6b7-4ec4-aa68-dae9378f7789"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:06:14.576601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.576577    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5hw2\" (UniqueName: \"kubernetes.io/projected/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-kube-api-access-q5hw2\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:14.576601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.576601    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:14.576740 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:14.576615    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:06:15.379320 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:15.379286    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"
Sep 12 02:06:15.400731 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:15.400701    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:06:15.405694 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:15.405669    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-98f68e/e2e-conflict-b-3901f64e-kserve-54bcfdc899-2jxwl"]
Sep 12 02:06:16.555577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:16.555547    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" path="/var/lib/kubelet/pods/d91ac3e7-d6b7-4ec4-aa68-dae9378f7789/volumes"
Sep 12 02:06:17.341021 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:17.340992    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:06:22.235330 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.235292    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:06:22.235776 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.235725    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" containerName="main"
Sep 12 02:06:22.235776 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.235739    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" containerName="main"
Sep 12 02:06:22.235888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.235803    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d91ac3e7-d6b7-4ec4-aa68-dae9378f7789" containerName="main"
Sep 12 02:06:22.239316 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.239297    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.241901 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.241878    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-60c2473b\"/\"e2e-lifecycle-model-60c2473b-kserve-self-signed-certs\""
Sep 12 02:06:22.243287 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.243266    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-60c2473b\"/\"openshift-service-ca.crt\""
Sep 12 02:06:22.243355 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.243294    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-60c2473b\"/\"default-dockercfg-c6tgn\""
Sep 12 02:06:22.243355 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.243333    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-60c2473b\"/\"kube-root-ca.crt\""
Sep 12 02:06:22.247243 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.247222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:06:22.347848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.347811    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.348063 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.347881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvzg\" (UniqueName: \"kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.348063 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.347943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.348063 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.347998    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.348252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.348133    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.348252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.348219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449693 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449657    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449759    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvzg\" (UniqueName: \"kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.449888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.449872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.450200 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.450177    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.450260 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.450247    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.450354 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.450332    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.452011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.451994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.452408 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.452389    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.458104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.458084    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvzg\" (UniqueName: \"kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg\") pod \"e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") " pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.549805 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.549710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:22.859062 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:22.858840    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:06:22.861923 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:06:22.861891    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d064bc2_0bce_41ef_8d65_3a974cd5df69.slice/crio-1756d74f2c44525f89f73f240f1603cb1092c4ac935290b91dfffc691fb6222d WatchSource:0}: Error finding container 1756d74f2c44525f89f73f240f1603cb1092c4ac935290b91dfffc691fb6222d: Status 404 returned error can't find the container with id 1756d74f2c44525f89f73f240f1603cb1092c4ac935290b91dfffc691fb6222d
Sep 12 02:06:23.423608 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:23.423568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" event={"ID":"0d064bc2-0bce-41ef-8d65-3a974cd5df69","Type":"ContainerStarted","Data":"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"}
Sep 12 02:06:23.423608 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:23.423608    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" event={"ID":"0d064bc2-0bce-41ef-8d65-3a974cd5df69","Type":"ContainerStarted","Data":"1756d74f2c44525f89f73f240f1603cb1092c4ac935290b91dfffc691fb6222d"}
Sep 12 02:06:23.444128 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:23.444082    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" podStartSLOduration=1.444067662 podStartE2EDuration="1.444067662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:06:23.440986479 +0000 UTC m=+1277.367932365" watchObservedRunningTime="2026-09-12 02:06:23.444067662 +0000 UTC m=+1277.371013548"
Sep 12 02:06:27.447117 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.447084    2582 generic.go:356] "Generic (PLEG): container finished" podID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerID="54e860885570ee9dd8e10da9a54c2ea22608bcb9b494c13dc83bef92424140ef" exitCode=0
Sep 12 02:06:27.447569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.447160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerDied","Data":"54e860885570ee9dd8e10da9a54c2ea22608bcb9b494c13dc83bef92424140ef"}
Sep 12 02:06:27.447614 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.447605    2582 scope.go:117] "RemoveContainer" containerID="54e860885570ee9dd8e10da9a54c2ea22608bcb9b494c13dc83bef92424140ef"
Sep 12 02:06:27.448806 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.448768    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:06:27.448908 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.448837    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:06:27.448908 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:27.448874    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:06:28.454699 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:28.454664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerStarted","Data":"c474a7e03a656f75a671ced4a7ea92da30ee4ff7f67689dd7a3ce13da1b54b2b"}
Sep 12 02:06:28.607814 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:28.607783    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:06:28.608940 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:28.608905    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:28.609104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:28.608974    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:29.288491 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:29.288443    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:06:29.608598 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:29.608518    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:29.608598 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:29.608570    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:30.608459 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:30.608419    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:30.608648 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:30.608487    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:31.608648 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:31.608607    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:31.609024 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:31.608669    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:32.555497 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:32.555469    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:32.555687 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:32.555509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:32.563218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:32.563195    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:32.608538 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:32.608502    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:32.608696 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:32.608563    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:33.505757 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:33.505718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:06:33.608543 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:33.608501    2582 patch_prober.go:28] interesting pod/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 12 02:06:33.608731 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:33.608560    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 12 02:06:34.612325 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:34.612293    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:06:34.613254 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:06:34.613232    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:07:31.086756 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.086571    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:07:31.087448 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.087387    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" podUID="f97df382-5559-4845-bff4-ad2da1e39431" containerName="payload-processing" containerID="cri-o://4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd" gracePeriod=30
Sep 12 02:07:31.357839 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.357508    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:07:31.508943 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.508903    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume\") pod \"f97df382-5559-4845-bff4-ad2da1e39431\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") "
Sep 12 02:07:31.509149 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.508998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jq7v7\" (UniqueName: \"kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7\") pod \"f97df382-5559-4845-bff4-ad2da1e39431\" (UID: \"f97df382-5559-4845-bff4-ad2da1e39431\") "
Sep 12 02:07:31.509277 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.509253    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f97df382-5559-4845-bff4-ad2da1e39431" (UID: "f97df382-5559-4845-bff4-ad2da1e39431"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:07:31.509427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.509411    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f97df382-5559-4845-bff4-ad2da1e39431-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:31.511190 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.511165    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7" (OuterVolumeSpecName: "kube-api-access-jq7v7") pod "f97df382-5559-4845-bff4-ad2da1e39431" (UID: "f97df382-5559-4845-bff4-ad2da1e39431"). InnerVolumeSpecName "kube-api-access-jq7v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:07:31.610245 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.610179    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq7v7\" (UniqueName: \"kubernetes.io/projected/f97df382-5559-4845-bff4-ad2da1e39431-kube-api-access-jq7v7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:31.785611 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.785577    2582 generic.go:356] "Generic (PLEG): container finished" podID="f97df382-5559-4845-bff4-ad2da1e39431" containerID="4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd" exitCode=0
Sep 12 02:07:31.785808 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.785617    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"
Sep 12 02:07:31.785808 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.785662    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" event={"ID":"f97df382-5559-4845-bff4-ad2da1e39431","Type":"ContainerDied","Data":"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"}
Sep 12 02:07:31.785808 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.785721    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht" event={"ID":"f97df382-5559-4845-bff4-ad2da1e39431","Type":"ContainerDied","Data":"79eb68f7000dcf50e54ca0ee2e3be6e8747979f9c63eccdc6ea8379257edb949"}
Sep 12 02:07:31.785808 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.785752    2582 scope.go:117] "RemoveContainer" containerID="4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"
Sep 12 02:07:31.795706 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.795684    2582 scope.go:117] "RemoveContainer" containerID="4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"
Sep 12 02:07:31.796006 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:07:31.795986    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd\": container with ID starting with 4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd not found: ID does not exist" containerID="4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"
Sep 12 02:07:31.796109 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.796014    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd"} err="failed to get container status \"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd\": rpc error: code = NotFound desc = could not find container \"4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd\": container with ID starting with 4e2593195330e6fb5acb7f2dc753818d7adec99db67da6045027123e787dc1fd not found: ID does not exist"
Sep 12 02:07:31.807371 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.807347    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:07:31.814268 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:31.814246    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-60c2473b-d46dffdd5-qjvht"]
Sep 12 02:07:32.557306 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:32.557277    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97df382-5559-4845-bff4-ad2da1e39431" path="/var/lib/kubelet/pods/f97df382-5559-4845-bff4-ad2da1e39431/volumes"
Sep 12 02:07:39.621517 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:39.621476    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:07:54.640328 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.640283    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" podUID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" containerName="main" containerID="cri-o://87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b" gracePeriod=45
Sep 12 02:07:54.896559 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.896501    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:07:54.910670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.910643    2582 generic.go:356] "Generic (PLEG): container finished" podID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" containerID="87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b" exitCode=0
Sep 12 02:07:54.910793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.910675    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"
Sep 12 02:07:54.910793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.910727    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" event={"ID":"0d064bc2-0bce-41ef-8d65-3a974cd5df69","Type":"ContainerDied","Data":"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"}
Sep 12 02:07:54.910793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.910779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9" event={"ID":"0d064bc2-0bce-41ef-8d65-3a974cd5df69","Type":"ContainerDied","Data":"1756d74f2c44525f89f73f240f1603cb1092c4ac935290b91dfffc691fb6222d"}
Sep 12 02:07:54.910969 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.910801    2582 scope.go:117] "RemoveContainer" containerID="87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"
Sep 12 02:07:54.920301 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.920217    2582 scope.go:117] "RemoveContainer" containerID="87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"
Sep 12 02:07:54.920700 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:07:54.920620    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b\": container with ID starting with 87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b not found: ID does not exist" containerID="87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"
Sep 12 02:07:54.920700 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.920659    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b"} err="failed to get container status \"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b\": rpc error: code = NotFound desc = could not find container \"87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b\": container with ID starting with 87802521d26d27e93e9ff539bf8d8338f3615bfc68f9af971a8d09de23af085b not found: ID does not exist"
Sep 12 02:07:54.931070 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931050    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931195 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931088    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931195 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljvzg\" (UniqueName: \"kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931315 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931198    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931315 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931315 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm\") pod \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\" (UID: \"0d064bc2-0bce-41ef-8d65-3a974cd5df69\") "
Sep 12 02:07:54.931477 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931357    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home" (OuterVolumeSpecName: "home") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:07:54.931477 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache" (OuterVolumeSpecName: "model-cache") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:07:54.931589 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931551    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:07:54.931693 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931674    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:54.931777 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931698    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:54.931777 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.931712    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:54.933517 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.933495    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg" (OuterVolumeSpecName: "kube-api-access-ljvzg") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "kube-api-access-ljvzg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:07:54.933719 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.933694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:07:54.933719 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:54.933702    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm" (OuterVolumeSpecName: "dshm") pod "0d064bc2-0bce-41ef-8d65-3a974cd5df69" (UID: "0d064bc2-0bce-41ef-8d65-3a974cd5df69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:07:55.032713 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:55.032673    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d064bc2-0bce-41ef-8d65-3a974cd5df69-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:55.032713 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:55.032713    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d064bc2-0bce-41ef-8d65-3a974cd5df69-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:55.032948 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:55.032727    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvzg\" (UniqueName: \"kubernetes.io/projected/0d064bc2-0bce-41ef-8d65-3a974cd5df69-kube-api-access-ljvzg\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:07:55.233360 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:55.233332    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:07:55.237229 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:55.237206    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-60c2473b/e2e-lifecycle-model-60c2473b-kserve-56466548c-2zhh9"]
Sep 12 02:07:56.555273 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:07:56.555245    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" path="/var/lib/kubelet/pods/0d064bc2-0bce-41ef-8d65-3a974cd5df69/volumes"
Sep 12 02:08:36.935445 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:36.935408    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:08:36.935992 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:36.935612    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy" containerID="cri-o://fb84daa7586ed6108cde0e4af39e288ad5f64e36d8a7e2765c05862ee2048bc7" gracePeriod=30
Sep 12 02:08:42.143624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.143590    2582 generic.go:356] "Generic (PLEG): container finished" podID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerID="fb84daa7586ed6108cde0e4af39e288ad5f64e36d8a7e2765c05862ee2048bc7" exitCode=0
Sep 12 02:08:42.144071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.143666    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerDied","Data":"fb84daa7586ed6108cde0e4af39e288ad5f64e36d8a7e2765c05862ee2048bc7"}
Sep 12 02:08:42.144071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.143748    2582 scope.go:117] "RemoveContainer" containerID="0faaf1eb01e17e74f78aec69b371bdc3f3450c66844d5a1365b6888b038c825a"
Sep 12 02:08:42.182950 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.182929    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:08:42.225312 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225234    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225312 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225276    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225312 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225302    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62crc\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225411    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225441    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:08:42.225870 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225551    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token\") pod \"09ffb94e-42e4-479e-8dc2-044fdd065761\" (UID: \"09ffb94e-42e4-479e-8dc2-044fdd065761\") "
Sep 12 02:08:42.225870 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.225754    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:08:42.226111 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.226086    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ffb94e-42e4-479e-8dc2-044fdd065761-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.226190 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.226114    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.226190 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.226181    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:08:42.226300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.226202    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:08:42.226300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.226238    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data" (OuterVolumeSpecName: "istio-data") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:08:42.227701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.227673    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token" (OuterVolumeSpecName: "istio-token") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:08:42.227962 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.227926    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:08:42.228082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.228059    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:08:42.228376 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.228358    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc" (OuterVolumeSpecName: "kube-api-access-62crc") pod "09ffb94e-42e4-479e-8dc2-044fdd065761" (UID: "09ffb94e-42e4-479e-8dc2-044fdd065761"). InnerVolumeSpecName "kube-api-access-62crc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:08:42.326918 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326895    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.326918 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326915    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.327071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326924    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62crc\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-kube-api-access-62crc\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.327071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326934    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.327071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326942    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.327071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326949    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ffb94e-42e4-479e-8dc2-044fdd065761-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:42.327071 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:42.326957    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ffb94e-42e4-479e-8dc2-044fdd065761-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:08:43.149814 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:43.149781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm" event={"ID":"09ffb94e-42e4-479e-8dc2-044fdd065761","Type":"ContainerDied","Data":"a0eebd95dc51a4e46987d2cc76f8d69c7fd862bc78363036813f25432f5107e7"}
Sep 12 02:08:43.149814 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:43.149811    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"
Sep 12 02:08:43.150257 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:43.149839    2582 scope.go:117] "RemoveContainer" containerID="fb84daa7586ed6108cde0e4af39e288ad5f64e36d8a7e2765c05862ee2048bc7"
Sep 12 02:08:43.167625 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:43.167603    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:08:43.170988 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:43.170966    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-60c2473b-openshift-default-5647554dd4-jhgqm"]
Sep 12 02:08:44.555057 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:44.555006    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" path="/var/lib/kubelet/pods/09ffb94e-42e4-479e-8dc2-044fdd065761/volumes"
Sep 12 02:08:48.424126 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424083    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424480    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424495    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97df382-5559-4845-bff4-ad2da1e39431" containerName="payload-processing"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424500    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97df382-5559-4845-bff4-ad2da1e39431" containerName="payload-processing"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424522    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" containerName="main"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424527    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" containerName="main"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424593    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:08:48.424597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424602    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:08:48.425115 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424607    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d064bc2-0bce-41ef-8d65-3a974cd5df69" containerName="main"
Sep 12 02:08:48.425115 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.424616    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97df382-5559-4845-bff4-ad2da1e39431" containerName="payload-processing"
Sep 12 02:08:48.429248 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.429224    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.432948 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.432919    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"kube-root-ca.crt\""
Sep 12 02:08:48.433080 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.432952    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"default-dockercfg-sh8kw\""
Sep 12 02:08:48.433080 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.432924    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"openshift-service-ca.crt\""
Sep 12 02:08:48.433080 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.433003    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f660a6\"/\"embedding-w0-f660a6-kserve-self-signed-certs\""
Sep 12 02:08:48.436656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.436619    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:08:48.477839 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.477809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkd24\" (UniqueName: \"kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.477947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.477876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.477993 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.477973    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.478032 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.478018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.478084 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.478064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.478126 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.478113    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.578809 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkd24\" (UniqueName: \"kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579000 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.578985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579364 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.579341    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579428 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.579364    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.579428 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.579385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.581160 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.581140    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.581337 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.581320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.586561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.586543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkd24\" (UniqueName: \"kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24\") pod \"embedding-w0-f660a6-kserve-54f8445c67-nlfcd\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") " pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:48.739201 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:48.739133    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:49.013076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:49.013030    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:08:49.014786 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:08:49.014753    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod113f3614_0d19_4194_94e0_6b35dd15a014.slice/crio-da7c037c405139ea5ce198beba97385eba2c7d7ea8d72325098f9c42072c10d3 WatchSource:0}: Error finding container da7c037c405139ea5ce198beba97385eba2c7d7ea8d72325098f9c42072c10d3: Status 404 returned error can't find the container with id da7c037c405139ea5ce198beba97385eba2c7d7ea8d72325098f9c42072c10d3
Sep 12 02:08:49.181250 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:49.181214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" event={"ID":"113f3614-0d19-4194-94e0-6b35dd15a014","Type":"ContainerStarted","Data":"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"}
Sep 12 02:08:49.181437 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:49.181265    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" event={"ID":"113f3614-0d19-4194-94e0-6b35dd15a014","Type":"ContainerStarted","Data":"da7c037c405139ea5ce198beba97385eba2c7d7ea8d72325098f9c42072c10d3"}
Sep 12 02:08:49.202406 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:49.202358    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" podStartSLOduration=1.202343345 podStartE2EDuration="1.202343345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:08:49.199594537 +0000 UTC m=+1423.126540422" watchObservedRunningTime="2026-09-12 02:08:49.202343345 +0000 UTC m=+1423.129289228"
Sep 12 02:08:58.739338 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:58.739301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:58.739813 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:58.739353    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:58.751543 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:58.751516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:08:59.245033 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:08:59.245005    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:09:31.348966 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.348935    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:09:31.349660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.349632    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:09:31.349660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.349658    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ffb94e-42e4-479e-8dc2-044fdd065761" containerName="istio-proxy"
Sep 12 02:09:31.353004 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.352983    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.355588 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.355567    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-caecee-openshift-default-dockercfg-sjbvd\""
Sep 12 02:09:31.364140 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.364111    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:09:31.376408 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376512 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376536    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376622 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376622 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376735 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376712    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvpv\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376769    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376838 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.376838 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.376828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.477794 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.477755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.477802    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.477832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.477883    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvpv\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.477923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478301 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478268    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478301 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478458 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478431    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478593 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478476    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478593 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478593 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478766 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.478982 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.478954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.479369 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.479347    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.481322 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.481301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.481427 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.481320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.487386 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.487360    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.487480 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.487405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvpv\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv\") pod \"e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") " pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.665416 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.665379    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:31.940910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.940834    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:09:31.944006 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:09:31.943977    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d6bf5_5b27_4ccf_864a_a8648727856e.slice/crio-c6795279d866d5a6126a12334efce9d4c68e96cd6bd4a795c6b4d6784ca89d5c WatchSource:0}: Error finding container c6795279d866d5a6126a12334efce9d4c68e96cd6bd4a795c6b4d6784ca89d5c: Status 404 returned error can't find the container with id c6795279d866d5a6126a12334efce9d4c68e96cd6bd4a795c6b4d6784ca89d5c
Sep 12 02:09:31.946395 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.946361    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:09:31.946470 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.946425    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:09:31.946509 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:31.946471    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:09:32.403255 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:32.403217    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerStarted","Data":"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"}
Sep 12 02:09:32.403705 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:32.403270    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerStarted","Data":"c6795279d866d5a6126a12334efce9d4c68e96cd6bd4a795c6b4d6784ca89d5c"}
Sep 12 02:09:32.424407 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:32.424356    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podStartSLOduration=1.424340403 podStartE2EDuration="1.424340403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:09:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:09:32.422500826 +0000 UTC m=+1466.349446714" watchObservedRunningTime="2026-09-12 02:09:32.424340403 +0000 UTC m=+1466.351286289"
Sep 12 02:09:32.665787 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:32.665705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:33.672009 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:33.671974    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:34.413563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:34.413522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:34.414406 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:34.414386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:09:38.756681 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.756640    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:09:38.769551 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.769465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.772006 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.771979    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:09:38.773505 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.773486    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-caecee-serving-cert\""
Sep 12 02:09:38.852826 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.852796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zl79\" (UniqueName: \"kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.853010 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.852856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.853113 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.853022    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.853179 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.853124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.953819 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.953784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.954022 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.953837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.954022 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.953868    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zl79\" (UniqueName: \"kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.954022 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.953923    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.954601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.954570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.956457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.956436    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.956553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.956480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:38.962890 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:38.962868    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zl79\" (UniqueName: \"kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79\") pod \"maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:39.084127 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.084029    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:39.362158 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.362087    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:09:39.365861 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.365838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.368393 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.368367    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-caecee-dockercfg-6lnws\""
Sep 12 02:09:39.368597 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.368575    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-caecee\""
Sep 12 02:09:39.368906 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.368884    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:09:39.370749 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:09:39.370722    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72af814_cdb2_4fd5_b6df_a2527d1819be.slice/crio-4e62fbd769e35e47f28c83cf1763c1d44243f33080ece71e4db76bd487bc0af2 WatchSource:0}: Error finding container 4e62fbd769e35e47f28c83cf1763c1d44243f33080ece71e4db76bd487bc0af2: Status 404 returned error can't find the container with id 4e62fbd769e35e47f28c83cf1763c1d44243f33080ece71e4db76bd487bc0af2
Sep 12 02:09:39.372662 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.372640    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:09:39.377581 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.377545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:09:39.442300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.442186    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" event={"ID":"b72af814-cdb2-4fd5-b6df-a2527d1819be","Type":"ContainerStarted","Data":"4e62fbd769e35e47f28c83cf1763c1d44243f33080ece71e4db76bd487bc0af2"}
Sep 12 02:09:39.459779 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.459530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.459779 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.459673    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rj6\" (UniqueName: \"kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.560408 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.560375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rj6\" (UniqueName: \"kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.560562 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.560438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.561166 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.561147    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.568094 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.568063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rj6\" (UniqueName: \"kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6\") pod \"payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") " pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.688685 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.688650    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:39.960991 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:39.960963    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:09:39.961949 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:09:39.961924    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f1ed73_07a0_4c35_a468_50e0e07e9523.slice/crio-af7a74df9d6b93e04f6973f1ac3cb5b89b8a76899c66e8d42dd13dfecc3633c2 WatchSource:0}: Error finding container af7a74df9d6b93e04f6973f1ac3cb5b89b8a76899c66e8d42dd13dfecc3633c2: Status 404 returned error can't find the container with id af7a74df9d6b93e04f6973f1ac3cb5b89b8a76899c66e8d42dd13dfecc3633c2
Sep 12 02:09:40.450409 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.450369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" event={"ID":"09f1ed73-07a0-4c35-a468-50e0e07e9523","Type":"ContainerStarted","Data":"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"}
Sep 12 02:09:40.450604 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.450428    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" event={"ID":"09f1ed73-07a0-4c35-a468-50e0e07e9523","Type":"ContainerStarted","Data":"af7a74df9d6b93e04f6973f1ac3cb5b89b8a76899c66e8d42dd13dfecc3633c2"}
Sep 12 02:09:40.450604 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.450509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:09:40.452084 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.452031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" event={"ID":"b72af814-cdb2-4fd5-b6df-a2527d1819be","Type":"ContainerStarted","Data":"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"}
Sep 12 02:09:40.452244 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.452153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:40.470635 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.470591    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" podStartSLOduration=1.4705791129999999 podStartE2EDuration="1.470579113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:09:40.468554105 +0000 UTC m=+1474.395499992" watchObservedRunningTime="2026-09-12 02:09:40.470579113 +0000 UTC m=+1474.397524999"
Sep 12 02:09:40.490859 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:40.490816    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" podStartSLOduration=2.048036999 podStartE2EDuration="2.490803567s" totalImagesPullingTime="442.766568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:09:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:09:40.489936588 +0000 UTC m=+1474.416882475" watchObservedRunningTime="2026-09-12 02:09:40.490803567 +0000 UTC m=+1474.417749452"
Sep 12 02:09:46.462729 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:46.462699    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:09:51.459213 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:09:51.459183    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:10:00.707411 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:00.707376    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d6bf5_5b27_4ccf_864a_a8648727856e.slice/crio-38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44d6bf5_5b27_4ccf_864a_a8648727856e.slice/crio-conmon-38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7.scope\": RecentStats: unable to find data in memory cache]"
Sep 12 02:10:01.566603 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.566569    2582 generic.go:356] "Generic (PLEG): container finished" podID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerID="38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7" exitCode=0
Sep 12 02:10:01.566794 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.566640    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerDied","Data":"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"}
Sep 12 02:10:01.567089 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.567075    2582 scope.go:117] "RemoveContainer" containerID="38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"
Sep 12 02:10:01.568323 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.568284    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:10:01.568426 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.568368    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:10:01.568426 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:01.568404    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:10:02.574399 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:02.574362    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerStarted","Data":"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"}
Sep 12 02:10:02.665846 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:02.665814    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:02.667127 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:02.667099    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:02.667263 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:02.667146    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:03.666601 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:03.666564    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:03.667007 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:03.666638    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:04.413762 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:04.413721    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:04.618189 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:04.618158    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:10:04.666411 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:04.666324    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:04.666411 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:04.666383    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:05.666143 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:05.666105    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:05.666524 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:05.666157    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:06.634782 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:06.634745    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:10:06.638326 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:06.638301    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:10:06.665994 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:06.665964    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:06.666106 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:06.666054    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:07.666059 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:07.666014    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:07.666437 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:07.666106    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:08.666413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:08.666378    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Sep 12 02:10:08.666903 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:08.666621    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Sep 12 02:10:09.671106 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:09.671075    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:09.672155 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:09.672136    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:14.382181 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.382149    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:10:14.385694 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.385676    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.388205 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.388181    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b23bee-dockercfg-7tqst\""
Sep 12 02:10:14.388332 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.388185    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b23bee\""
Sep 12 02:10:14.398377 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.398347    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:10:14.517633 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.517599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwk9q\" (UniqueName: \"kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.517633 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.517637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.618600 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.618568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwk9q\" (UniqueName: \"kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.618600 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.618604    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.619266 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.619246    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.626807 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.626779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwk9q\" (UniqueName: \"kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q\") pod \"payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") " pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.697182 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.697111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:14.967417 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:14.967389    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:10:14.968909 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:10:14.968878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc027080d_04c5_4a2a_bd7a_b7f1b34e19ed.slice/crio-28e47e4cfdcd48dc6e73096bc8537b2c2b23afd2be09d44ed41a811215d972f0 WatchSource:0}: Error finding container 28e47e4cfdcd48dc6e73096bc8537b2c2b23afd2be09d44ed41a811215d972f0: Status 404 returned error can't find the container with id 28e47e4cfdcd48dc6e73096bc8537b2c2b23afd2be09d44ed41a811215d972f0
Sep 12 02:10:15.661548 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:15.661515    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" event={"ID":"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed","Type":"ContainerStarted","Data":"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"}
Sep 12 02:10:15.662023 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:15.661573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" event={"ID":"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed","Type":"ContainerStarted","Data":"28e47e4cfdcd48dc6e73096bc8537b2c2b23afd2be09d44ed41a811215d972f0"}
Sep 12 02:10:15.662023 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:15.661644    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:15.678147 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:15.678096    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" podStartSLOduration=1.678077901 podStartE2EDuration="1.678077901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:10:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:10:15.676376274 +0000 UTC m=+1509.603322160" watchObservedRunningTime="2026-09-12 02:10:15.678077901 +0000 UTC m=+1509.605023788"
Sep 12 02:10:18.424926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.424578    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 02:10:18.424926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.424817    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" podUID="152472a3-44b0-4843-ba42-2ee526496455" containerName="maas-api" containerID="cri-o://3390552ff99c0d3fbc9c5d2a2920d1e4764eaeb7d3d400b9c28716257a65121f" gracePeriod=30
Sep 12 02:10:18.477877 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.477844    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 02:10:18.478308 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.478117    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" podUID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" containerName="payload-processing" containerID="cri-o://5c0bf5230a5b5c4b795895f333fc5d0f272de4ab318df3e457f733deebbbfe6e" gracePeriod=30
Sep 12 02:10:18.691452 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.691364    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" containerID="5c0bf5230a5b5c4b795895f333fc5d0f272de4ab318df3e457f733deebbbfe6e" exitCode=0
Sep 12 02:10:18.691653 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.691410    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" event={"ID":"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa","Type":"ContainerDied","Data":"5c0bf5230a5b5c4b795895f333fc5d0f272de4ab318df3e457f733deebbbfe6e"}
Sep 12 02:10:18.693886 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.693861    2582 generic.go:356] "Generic (PLEG): container finished" podID="152472a3-44b0-4843-ba42-2ee526496455" containerID="3390552ff99c0d3fbc9c5d2a2920d1e4764eaeb7d3d400b9c28716257a65121f" exitCode=0
Sep 12 02:10:18.694025 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.693962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" event={"ID":"152472a3-44b0-4843-ba42-2ee526496455","Type":"ContainerDied","Data":"3390552ff99c0d3fbc9c5d2a2920d1e4764eaeb7d3d400b9c28716257a65121f"}
Sep 12 02:10:18.782924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.782892    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 02:10:18.810250 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.810225    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 02:10:18.856008 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.855979    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca\") pod \"152472a3-44b0-4843-ba42-2ee526496455\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") "
Sep 12 02:10:18.856177 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.856020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vz5fd\" (UniqueName: \"kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd\") pod \"152472a3-44b0-4843-ba42-2ee526496455\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") "
Sep 12 02:10:18.856177 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.856086    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs\") pod \"152472a3-44b0-4843-ba42-2ee526496455\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") "
Sep 12 02:10:18.856177 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.856159    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls\") pod \"152472a3-44b0-4843-ba42-2ee526496455\" (UID: \"152472a3-44b0-4843-ba42-2ee526496455\") "
Sep 12 02:10:18.856416 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.856392    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "152472a3-44b0-4843-ba42-2ee526496455" (UID: "152472a3-44b0-4843-ba42-2ee526496455"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:18.856522 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.856503    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/152472a3-44b0-4843-ba42-2ee526496455-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:18.858208 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.858186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd" (OuterVolumeSpecName: "kube-api-access-vz5fd") pod "152472a3-44b0-4843-ba42-2ee526496455" (UID: "152472a3-44b0-4843-ba42-2ee526496455"). InnerVolumeSpecName "kube-api-access-vz5fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:18.858208 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.858186    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "152472a3-44b0-4843-ba42-2ee526496455" (UID: "152472a3-44b0-4843-ba42-2ee526496455"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:10:18.858337 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.858206    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "152472a3-44b0-4843-ba42-2ee526496455" (UID: "152472a3-44b0-4843-ba42-2ee526496455"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:10:18.957426 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume\") pod \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") "
Sep 12 02:10:18.957426 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm64j\" (UniqueName: \"kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j\") pod \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\" (UID: \"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa\") "
Sep 12 02:10:18.957649 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957620    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz5fd\" (UniqueName: \"kubernetes.io/projected/152472a3-44b0-4843-ba42-2ee526496455-kube-api-access-vz5fd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:18.957649 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957632    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:18.957649 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957641    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/152472a3-44b0-4843-ba42-2ee526496455-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:18.957797 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.957692    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" (UID: "ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:18.959392 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:18.959369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j" (OuterVolumeSpecName: "kube-api-access-xm64j") pod "ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" (UID: "ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa"). InnerVolumeSpecName "kube-api-access-xm64j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:19.058313 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.058281    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:19.058313 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.058309    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm64j\" (UniqueName: \"kubernetes.io/projected/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa-kube-api-access-xm64j\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:19.637848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.637810    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" podUID="113f3614-0d19-4194-94e0-6b35dd15a014" containerName="main" containerID="cri-o://cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c" gracePeriod=45
Sep 12 02:10:19.700351 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.700284    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"
Sep 12 02:10:19.700351 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.700298    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8" event={"ID":"ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa","Type":"ContainerDied","Data":"20524fc8a643e0c16dde537e41e6e16560a89ddff8baaa95b7095f5ad265bc21"}
Sep 12 02:10:19.700544 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.700369    2582 scope.go:117] "RemoveContainer" containerID="5c0bf5230a5b5c4b795895f333fc5d0f272de4ab318df3e457f733deebbbfe6e"
Sep 12 02:10:19.701777 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.701757    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c" event={"ID":"152472a3-44b0-4843-ba42-2ee526496455","Type":"ContainerDied","Data":"d20e64fe2e5d7a5376152a65e8c4f546db072c198fea325360b6234ac77bb2b2"}
Sep 12 02:10:19.701864 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.701783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"
Sep 12 02:10:19.724551 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.724530    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 02:10:19.734134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.734111    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-f660a6-788d5ff677-lq8z8"]
Sep 12 02:10:19.746873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.746841    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 02:10:19.748868 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.748845    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-f660a6-5685d56bc9-82z5c"]
Sep 12 02:10:19.749717 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.749702    2582 scope.go:117] "RemoveContainer" containerID="3390552ff99c0d3fbc9c5d2a2920d1e4764eaeb7d3d400b9c28716257a65121f"
Sep 12 02:10:19.893239 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.893216    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:10:19.967251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967174    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967263    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967351    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkd24\" (UniqueName: \"kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967387    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967665 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs\") pod \"113f3614-0d19-4194-94e0-6b35dd15a014\" (UID: \"113f3614-0d19-4194-94e0-6b35dd15a014\") "
Sep 12 02:10:19.967665 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home" (OuterVolumeSpecName: "home") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:19.967665 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967621    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:19.967795 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967679    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache" (OuterVolumeSpecName: "model-cache") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:19.967795 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967781    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-home\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:19.967909 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967804    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-tmp-dir\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:19.967909 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.967818    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-model-cache\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:19.969919 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.969885    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm" (OuterVolumeSpecName: "dshm") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:19.969919 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.969893    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24" (OuterVolumeSpecName: "kube-api-access-hkd24") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "kube-api-access-hkd24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:19.970118 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:19.969961    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "113f3614-0d19-4194-94e0-6b35dd15a014" (UID: "113f3614-0d19-4194-94e0-6b35dd15a014"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:10:20.068457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.068424    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/113f3614-0d19-4194-94e0-6b35dd15a014-tls-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:20.068457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.068447    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/113f3614-0d19-4194-94e0-6b35dd15a014-dshm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:20.068457 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.068457    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkd24\" (UniqueName: \"kubernetes.io/projected/113f3614-0d19-4194-94e0-6b35dd15a014-kube-api-access-hkd24\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:20.556337 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.556304    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="152472a3-44b0-4843-ba42-2ee526496455" path="/var/lib/kubelet/pods/152472a3-44b0-4843-ba42-2ee526496455/volumes"
Sep 12 02:10:20.556704 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.556690    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" path="/var/lib/kubelet/pods/ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa/volumes"
Sep 12 02:10:20.709561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.709527    2582 generic.go:356] "Generic (PLEG): container finished" podID="113f3614-0d19-4194-94e0-6b35dd15a014" containerID="cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c" exitCode=0
Sep 12 02:10:20.709980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.709573    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"
Sep 12 02:10:20.709980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.709615    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" event={"ID":"113f3614-0d19-4194-94e0-6b35dd15a014","Type":"ContainerDied","Data":"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"}
Sep 12 02:10:20.709980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.709664    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd" event={"ID":"113f3614-0d19-4194-94e0-6b35dd15a014","Type":"ContainerDied","Data":"da7c037c405139ea5ce198beba97385eba2c7d7ea8d72325098f9c42072c10d3"}
Sep 12 02:10:20.709980 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.709683    2582 scope.go:117] "RemoveContainer" containerID="cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"
Sep 12 02:10:20.718952 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.718934    2582 scope.go:117] "RemoveContainer" containerID="cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"
Sep 12 02:10:20.719255 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:20.719232    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c\": container with ID starting with cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c not found: ID does not exist" containerID="cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"
Sep 12 02:10:20.719321 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.719265    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c"} err="failed to get container status \"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c\": rpc error: code = NotFound desc = could not find container \"cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c\": container with ID starting with cfdc1b1867dd58afc3b04d7e9b71d5234de9d9bdb65bcbb830d1b7f56808df6c not found: ID does not exist"
Sep 12 02:10:20.729274 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.729251    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:10:20.733348 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:20.733325    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f660a6/embedding-w0-f660a6-kserve-54f8445c67-nlfcd"]
Sep 12 02:10:22.555727 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:22.555695    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113f3614-0d19-4194-94e0-6b35dd15a014" path="/var/lib/kubelet/pods/113f3614-0d19-4194-94e0-6b35dd15a014/volumes"
Sep 12 02:10:26.204291 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:26.204257    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 02:10:26.204817 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:26.204768    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" containerID="cri-o://102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be" gracePeriod=30
Sep 12 02:10:26.668579 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:26.668549    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:10:30.977069 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:30.976960    2582 patch_prober.go:28] interesting pod/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:10:30.977069 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:30.977014    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:10:31.488892 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.488836    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 02:10:31.581926 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.581898    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.581940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.581965    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.581992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582064    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhpj8\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582403 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582193    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582403 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data" (OuterVolumeSpecName: "istio-data") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:31.582403 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582231    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582403 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582258    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket\") pod \"633d9bc5-d002-488f-80c6-c9f974b0de35\" (UID: \"633d9bc5-d002-488f-80c6-c9f974b0de35\") "
Sep 12 02:10:31.582403 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:31.582668 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582519    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:31.582668 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:31.582668 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582644    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:31.582668 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582659    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.582873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582682    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.582873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.582698    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.590928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.585170    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:10:31.590928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.585193    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token" (OuterVolumeSpecName: "istio-token") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:31.590928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.585171    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:31.590928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.585212    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8" (OuterVolumeSpecName: "kube-api-access-xhpj8") pod "633d9bc5-d002-488f-80c6-c9f974b0de35" (UID: "633d9bc5-d002-488f-80c6-c9f974b0de35"). InnerVolumeSpecName "kube-api-access-xhpj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:31.683503 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683475    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.683503 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683503    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.683664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683517    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhpj8\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-kube-api-access-xhpj8\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.683664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683532    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/633d9bc5-d002-488f-80c6-c9f974b0de35-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.683664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683544    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/633d9bc5-d002-488f-80c6-c9f974b0de35-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.683664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.683556    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/633d9bc5-d002-488f-80c6-c9f974b0de35-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:31.780900 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.780827    2582 generic.go:356] "Generic (PLEG): container finished" podID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerID="102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be" exitCode=0
Sep 12 02:10:31.780900 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.780886    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"
Sep 12 02:10:31.781155 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.780917    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerDied","Data":"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"}
Sep 12 02:10:31.781155 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.781015    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm" event={"ID":"633d9bc5-d002-488f-80c6-c9f974b0de35","Type":"ContainerDied","Data":"2104784e2c076a5213b49f2f40a86fe2e039cd9756f2b62e80e56d382564ec70"}
Sep 12 02:10:31.781155 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.781075    2582 scope.go:117] "RemoveContainer" containerID="102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"
Sep 12 02:10:31.790537 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.790517    2582 scope.go:117] "RemoveContainer" containerID="05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"
Sep 12 02:10:31.798589 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.798570    2582 scope.go:117] "RemoveContainer" containerID="102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"
Sep 12 02:10:31.798858 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:31.798839    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be\": container with ID starting with 102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be not found: ID does not exist" containerID="102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"
Sep 12 02:10:31.798925 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.798865    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be"} err="failed to get container status \"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be\": rpc error: code = NotFound desc = could not find container \"102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be\": container with ID starting with 102dfac8f031f0242fa83a0955adb862f40c137d273df38d9976796dc2ec79be not found: ID does not exist"
Sep 12 02:10:31.798925 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.798881    2582 scope.go:117] "RemoveContainer" containerID="05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"
Sep 12 02:10:31.799167 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:31.799152    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588\": container with ID starting with 05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588 not found: ID does not exist" containerID="05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"
Sep 12 02:10:31.799217 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.799171    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588"} err="failed to get container status \"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588\": rpc error: code = NotFound desc = could not find container \"05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588\": container with ID starting with 05fcc88351e7a692e804fd3b83ddf4289635732e6992e95d206a7e20c1b9a588 not found: ID does not exist"
Sep 12 02:10:31.803020 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.802998    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 02:10:31.808551 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:31.808530    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f660a6-openshift-default-86c64c5854-9bdtm"]
Sep 12 02:10:32.555593 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:32.555559    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" path="/var/lib/kubelet/pods/633d9bc5-d002-488f-80c6-c9f974b0de35/volumes"
Sep 12 02:10:46.988865 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:46.988826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:10:46.989417 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:46.989092    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" podUID="b72af814-cdb2-4fd5-b6df-a2527d1819be" containerName="maas-api" containerID="cri-o://4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431" gracePeriod=30
Sep 12 02:10:47.061417 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.061172    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:10:47.062240 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.061978    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" podUID="09f1ed73-07a0-4c35-a468-50e0e07e9523" containerName="payload-processing" containerID="cri-o://fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3" gracePeriod=30
Sep 12 02:10:47.320897 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.319221    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:10:47.367580 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.367554    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:10:47.430459 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs\") pod \"b72af814-cdb2-4fd5-b6df-a2527d1819be\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") "
Sep 12 02:10:47.430645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls\") pod \"b72af814-cdb2-4fd5-b6df-a2527d1819be\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") "
Sep 12 02:10:47.430645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430577    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume\") pod \"09f1ed73-07a0-4c35-a468-50e0e07e9523\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") "
Sep 12 02:10:47.430645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2rj6\" (UniqueName: \"kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6\") pod \"09f1ed73-07a0-4c35-a468-50e0e07e9523\" (UID: \"09f1ed73-07a0-4c35-a468-50e0e07e9523\") "
Sep 12 02:10:47.430645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zl79\" (UniqueName: \"kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79\") pod \"b72af814-cdb2-4fd5-b6df-a2527d1819be\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") "
Sep 12 02:10:47.430859 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430674    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca\") pod \"b72af814-cdb2-4fd5-b6df-a2527d1819be\" (UID: \"b72af814-cdb2-4fd5-b6df-a2527d1819be\") "
Sep 12 02:10:47.431055 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.430984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "09f1ed73-07a0-4c35-a468-50e0e07e9523" (UID: "09f1ed73-07a0-4c35-a468-50e0e07e9523"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:47.431376 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.431349    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b72af814-cdb2-4fd5-b6df-a2527d1819be" (UID: "b72af814-cdb2-4fd5-b6df-a2527d1819be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:47.432818 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.432789    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b72af814-cdb2-4fd5-b6df-a2527d1819be" (UID: "b72af814-cdb2-4fd5-b6df-a2527d1819be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:10:47.433234 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.433204    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79" (OuterVolumeSpecName: "kube-api-access-6zl79") pod "b72af814-cdb2-4fd5-b6df-a2527d1819be" (UID: "b72af814-cdb2-4fd5-b6df-a2527d1819be"). InnerVolumeSpecName "kube-api-access-6zl79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:47.433345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.433314    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b72af814-cdb2-4fd5-b6df-a2527d1819be" (UID: "b72af814-cdb2-4fd5-b6df-a2527d1819be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:10:47.433345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.433321    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6" (OuterVolumeSpecName: "kube-api-access-w2rj6") pod "09f1ed73-07a0-4c35-a468-50e0e07e9523" (UID: "09f1ed73-07a0-4c35-a468-50e0e07e9523"). InnerVolumeSpecName "kube-api-access-w2rj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531774    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531802    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f1ed73-07a0-4c35-a468-50e0e07e9523-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531812    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rj6\" (UniqueName: \"kubernetes.io/projected/09f1ed73-07a0-4c35-a468-50e0e07e9523-kube-api-access-w2rj6\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531822    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zl79\" (UniqueName: \"kubernetes.io/projected/b72af814-cdb2-4fd5-b6df-a2527d1819be-kube-api-access-6zl79\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531831    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b72af814-cdb2-4fd5-b6df-a2527d1819be-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.531858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.531839    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72af814-cdb2-4fd5-b6df-a2527d1819be-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:47.866200 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.866122    2582 generic.go:356] "Generic (PLEG): container finished" podID="b72af814-cdb2-4fd5-b6df-a2527d1819be" containerID="4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431" exitCode=0
Sep 12 02:10:47.866200 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.866163    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"
Sep 12 02:10:47.866418 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.866209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" event={"ID":"b72af814-cdb2-4fd5-b6df-a2527d1819be","Type":"ContainerDied","Data":"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"}
Sep 12 02:10:47.866418 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.866262    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8" event={"ID":"b72af814-cdb2-4fd5-b6df-a2527d1819be","Type":"ContainerDied","Data":"4e62fbd769e35e47f28c83cf1763c1d44243f33080ece71e4db76bd487bc0af2"}
Sep 12 02:10:47.866418 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.866294    2582 scope.go:117] "RemoveContainer" containerID="4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"
Sep 12 02:10:47.867582 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.867557    2582 generic.go:356] "Generic (PLEG): container finished" podID="09f1ed73-07a0-4c35-a468-50e0e07e9523" containerID="fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3" exitCode=0
Sep 12 02:10:47.867712 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.867592    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" event={"ID":"09f1ed73-07a0-4c35-a468-50e0e07e9523","Type":"ContainerDied","Data":"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"}
Sep 12 02:10:47.867712 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.867609    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"
Sep 12 02:10:47.867712 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.867638    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk" event={"ID":"09f1ed73-07a0-4c35-a468-50e0e07e9523","Type":"ContainerDied","Data":"af7a74df9d6b93e04f6973f1ac3cb5b89b8a76899c66e8d42dd13dfecc3633c2"}
Sep 12 02:10:47.876461 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.876440    2582 scope.go:117] "RemoveContainer" containerID="4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"
Sep 12 02:10:47.876724 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:47.876704    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431\": container with ID starting with 4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431 not found: ID does not exist" containerID="4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"
Sep 12 02:10:47.876801 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.876737    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431"} err="failed to get container status \"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431\": rpc error: code = NotFound desc = could not find container \"4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431\": container with ID starting with 4a71a4e45affa1f58b97e121ca09c82e70ae58b8eb0ed89a02538a4d5d563431 not found: ID does not exist"
Sep 12 02:10:47.876801 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.876764    2582 scope.go:117] "RemoveContainer" containerID="fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"
Sep 12 02:10:47.884326 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.884309    2582 scope.go:117] "RemoveContainer" containerID="fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"
Sep 12 02:10:47.884606 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:47.884586    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3\": container with ID starting with fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3 not found: ID does not exist" containerID="fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"
Sep 12 02:10:47.884709 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.884610    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3"} err="failed to get container status \"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3\": rpc error: code = NotFound desc = could not find container \"fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3\": container with ID starting with fc797cbab70d40e40f3808d8c9e43178a83c7ee679214496eb2a0590e8e525a3 not found: ID does not exist"
Sep 12 02:10:47.889078 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.889031    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:10:47.892732 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.892713    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-caecee-5fc5ff5898-4qvr8"]
Sep 12 02:10:47.903053 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.903010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:10:47.906799 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:47.906776    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-caecee-6fb985594c-2sbxk"]
Sep 12 02:10:48.555996 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:48.555965    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09f1ed73-07a0-4c35-a468-50e0e07e9523" path="/var/lib/kubelet/pods/09f1ed73-07a0-4c35-a468-50e0e07e9523/volumes"
Sep 12 02:10:48.556426 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:48.556410    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b72af814-cdb2-4fd5-b6df-a2527d1819be" path="/var/lib/kubelet/pods/b72af814-cdb2-4fd5-b6df-a2527d1819be/volumes"
Sep 12 02:10:54.614534 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:54.614500    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:10:54.615077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:54.614695    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" containerID="cri-o://b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4" gracePeriod=30
Sep 12 02:10:54.673344 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:54.673308    2582 patch_prober.go:28] interesting pod/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:10:54.673494 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:54.673374    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:10:59.856061 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.856023    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:59.930456 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.930384    2582 generic.go:356] "Generic (PLEG): container finished" podID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerID="b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4" exitCode=0
Sep 12 02:10:59.930456 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.930428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"
Sep 12 02:10:59.930616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.930472    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerDied","Data":"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"}
Sep 12 02:10:59.930616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.930532    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl" event={"ID":"b44d6bf5-5b27-4ccf-864a-a8648727856e","Type":"ContainerDied","Data":"c6795279d866d5a6126a12334efce9d4c68e96cd6bd4a795c6b4d6784ca89d5c"}
Sep 12 02:10:59.930616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.930556    2582 scope.go:117] "RemoveContainer" containerID="b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"
Sep 12 02:10:59.939327 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.939312    2582 scope.go:117] "RemoveContainer" containerID="38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"
Sep 12 02:10:59.945896 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.945873    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.945914    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.945958    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.945988    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946006    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946270 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946079    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cvpv\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946365 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:59.946568 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:59.946691 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946691 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946691 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946662    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket\") pod \"b44d6bf5-5b27-4ccf-864a-a8648727856e\" (UID: \"b44d6bf5-5b27-4ccf-864a-a8648727856e\") "
Sep 12 02:10:59.946896 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946587    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data" (OuterVolumeSpecName: "istio-data") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:59.946896 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.946870    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:10:59.947125 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.947033    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:59.947218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.947148    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:59.947218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.947168    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:59.947218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.947181    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:59.947218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.947193    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b44d6bf5-5b27-4ccf-864a-a8648727856e-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:10:59.948494 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.948461    2582 scope.go:117] "RemoveContainer" containerID="b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"
Sep 12 02:10:59.948882 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.948816    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:10:59.948882 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.948813    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:10:59.948882 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:59.948822    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4\": container with ID starting with b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4 not found: ID does not exist" containerID="b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"
Sep 12 02:10:59.948882 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.948871    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4"} err="failed to get container status \"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4\": rpc error: code = NotFound desc = could not find container \"b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4\": container with ID starting with b6c03d2525ffb319e6c6d4212033b84deb714796c489b6ca997ad4ec2620ddb4 not found: ID does not exist"
Sep 12 02:10:59.948882 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.948896    2582 scope.go:117] "RemoveContainer" containerID="38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"
Sep 12 02:10:59.949247 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.949089    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:10:59.949305 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:10:59.949234    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7\": container with ID starting with 38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7 not found: ID does not exist" containerID="38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"
Sep 12 02:10:59.949305 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.949267    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7"} err="failed to get container status \"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7\": rpc error: code = NotFound desc = could not find container \"38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7\": container with ID starting with 38769543f6c9d328d8a5dc27c1cc428861dbf07483c14c6714e4f42e24bfbbb7 not found: ID does not exist"
Sep 12 02:10:59.949383 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:10:59.949311    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv" (OuterVolumeSpecName: "kube-api-access-5cvpv") pod "b44d6bf5-5b27-4ccf-864a-a8648727856e" (UID: "b44d6bf5-5b27-4ccf-864a-a8648727856e"). InnerVolumeSpecName "kube-api-access-5cvpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:11:00.048631 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.048599    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:00.048631 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.048625    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:00.048631 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.048634    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:00.048829 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.048642    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b44d6bf5-5b27-4ccf-864a-a8648727856e-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:00.048829 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.048651    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvpv\" (UniqueName: \"kubernetes.io/projected/b44d6bf5-5b27-4ccf-864a-a8648727856e-kube-api-access-5cvpv\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:00.252431 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.252400    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:11:00.256893 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.256861    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-caecee-openshift-default-6c9d74489b-9z2kl"]
Sep 12 02:11:00.555749 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:00.555663    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" path="/var/lib/kubelet/pods/b44d6bf5-5b27-4ccf-864a-a8648727856e/volumes"
Sep 12 02:11:06.707027 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:06.706994    2582 scope.go:117] "RemoveContainer" containerID="8aeb6d21dd02bc90a0a0d76dfe143cac7cb5a509eeaa81825f503c5691866c81"
Sep 12 02:11:10.646200 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.646160    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:11:10.646687 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.646503    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" podUID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" containerName="payload-processing" containerID="cri-o://abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d" gracePeriod=30
Sep 12 02:11:10.911184 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.911095    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:11:10.987303 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.987271    2582 generic.go:356] "Generic (PLEG): container finished" podID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" containerID="abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d" exitCode=0
Sep 12 02:11:10.987472 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.987346    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"
Sep 12 02:11:10.987472 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.987349    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" event={"ID":"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed","Type":"ContainerDied","Data":"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"}
Sep 12 02:11:10.987549 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.987487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb" event={"ID":"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed","Type":"ContainerDied","Data":"28e47e4cfdcd48dc6e73096bc8537b2c2b23afd2be09d44ed41a811215d972f0"}
Sep 12 02:11:10.987549 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.987508    2582 scope.go:117] "RemoveContainer" containerID="abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"
Sep 12 02:11:10.997821 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.997796    2582 scope.go:117] "RemoveContainer" containerID="abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"
Sep 12 02:11:10.998154 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:11:10.998127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d\": container with ID starting with abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d not found: ID does not exist" containerID="abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"
Sep 12 02:11:10.998235 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:10.998167    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d"} err="failed to get container status \"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d\": rpc error: code = NotFound desc = could not find container \"abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d\": container with ID starting with abc128808d0703902968d8875242f58c48fe6ec055612403172967d33f033b7d not found: ID does not exist"
Sep 12 02:11:11.047799 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.047774    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume\") pod \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") "
Sep 12 02:11:11.047943 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.047892    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwk9q\" (UniqueName: \"kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q\") pod \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\" (UID: \"c027080d-04c5-4a2a-bd7a-b7f1b34e19ed\") "
Sep 12 02:11:11.048177 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.048157    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" (UID: "c027080d-04c5-4a2a-bd7a-b7f1b34e19ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:11:11.049895 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.049877    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q" (OuterVolumeSpecName: "kube-api-access-bwk9q") pod "c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" (UID: "c027080d-04c5-4a2a-bd7a-b7f1b34e19ed"). InnerVolumeSpecName "kube-api-access-bwk9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:11:11.149284 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.149250    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwk9q\" (UniqueName: \"kubernetes.io/projected/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-kube-api-access-bwk9q\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:11.149284 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.149276    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:11.309183 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.309150    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:11:11.311555 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:11.311526    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b23bee-85486c6f9-h4svb"]
Sep 12 02:11:12.557682 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:12.557647    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" path="/var/lib/kubelet/pods/c027080d-04c5-4a2a-bd7a-b7f1b34e19ed/volumes"
Sep 12 02:11:28.731566 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:28.731533    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:11:28.732179 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:28.731755    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" podUID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" containerName="payload-processing" containerID="cri-o://3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9" gracePeriod=30
Sep 12 02:11:29.008227 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.008202    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:11:29.074520 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.074486    2582 generic.go:356] "Generic (PLEG): container finished" podID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" containerID="3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9" exitCode=0
Sep 12 02:11:29.074701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.074526    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" event={"ID":"736c0762-6a3a-45fd-941a-aa2c0cd6d007","Type":"ContainerDied","Data":"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"}
Sep 12 02:11:29.074701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.074531    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"
Sep 12 02:11:29.074701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.074574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d" event={"ID":"736c0762-6a3a-45fd-941a-aa2c0cd6d007","Type":"ContainerDied","Data":"be2d470c7a9a35861b0443523f0fe3bf7cfd27640b7dc0e10a60af41b559d75e"}
Sep 12 02:11:29.074701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.074599    2582 scope.go:117] "RemoveContainer" containerID="3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"
Sep 12 02:11:29.084835 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.084814    2582 scope.go:117] "RemoveContainer" containerID="3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"
Sep 12 02:11:29.085127 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:11:29.085105    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9\": container with ID starting with 3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9 not found: ID does not exist" containerID="3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"
Sep 12 02:11:29.085213 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.085135    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9"} err="failed to get container status \"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9\": rpc error: code = NotFound desc = could not find container \"3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9\": container with ID starting with 3183243ab83e3aab6e72842da3cc68835144b975bd1ef028dd2315b828d7b3d9 not found: ID does not exist"
Sep 12 02:11:29.107572 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.107543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume\") pod \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") "
Sep 12 02:11:29.107726 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.107607    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gchkj\" (UniqueName: \"kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj\") pod \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\" (UID: \"736c0762-6a3a-45fd-941a-aa2c0cd6d007\") "
Sep 12 02:11:29.108105 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.108075    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "736c0762-6a3a-45fd-941a-aa2c0cd6d007" (UID: "736c0762-6a3a-45fd-941a-aa2c0cd6d007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:11:29.109667 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.109644    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj" (OuterVolumeSpecName: "kube-api-access-gchkj") pod "736c0762-6a3a-45fd-941a-aa2c0cd6d007" (UID: "736c0762-6a3a-45fd-941a-aa2c0cd6d007"). InnerVolumeSpecName "kube-api-access-gchkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:11:29.208839 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.208807    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchkj\" (UniqueName: \"kubernetes.io/projected/736c0762-6a3a-45fd-941a-aa2c0cd6d007-kube-api-access-gchkj\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:29.208839 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.208832    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/736c0762-6a3a-45fd-941a-aa2c0cd6d007-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:29.396188 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.396157    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:11:29.403582 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:29.403558    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-e8046f-7c4d685d87-wmk5d"]
Sep 12 02:11:30.561585 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:30.561551    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" path="/var/lib/kubelet/pods/736c0762-6a3a-45fd-941a-aa2c0cd6d007/volumes"
Sep 12 02:11:43.994956 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:43.994911    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 02:11:43.995798 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:43.995726    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" podUID="ca9fc4ef-79c2-4016-881d-599a91f818e1" containerName="maas-api" containerID="cri-o://bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a" gracePeriod=30
Sep 12 02:11:44.059645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.059609    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 02:11:44.059955 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.059903    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" podUID="eec9c7d8-8e68-42f5-953a-07488d2b7504" containerName="payload-pre-processing" containerID="cri-o://1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf" gracePeriod=30
Sep 12 02:11:44.340648 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.340616    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 02:11:44.438568 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.438535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rwhd\" (UniqueName: \"kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd\") pod \"eec9c7d8-8e68-42f5-953a-07488d2b7504\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") "
Sep 12 02:11:44.438568 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.438574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume\") pod \"eec9c7d8-8e68-42f5-953a-07488d2b7504\" (UID: \"eec9c7d8-8e68-42f5-953a-07488d2b7504\") "
Sep 12 02:11:44.439107 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.439075    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eec9c7d8-8e68-42f5-953a-07488d2b7504" (UID: "eec9c7d8-8e68-42f5-953a-07488d2b7504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:11:44.440703 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.440681    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd" (OuterVolumeSpecName: "kube-api-access-6rwhd") pod "eec9c7d8-8e68-42f5-953a-07488d2b7504" (UID: "eec9c7d8-8e68-42f5-953a-07488d2b7504"). InnerVolumeSpecName "kube-api-access-6rwhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:11:44.460741 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.460721    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 02:11:44.540006 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.539922    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncz94\" (UniqueName: \"kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94\") pod \"ca9fc4ef-79c2-4016-881d-599a91f818e1\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") "
Sep 12 02:11:44.540006 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.539966    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca\") pod \"ca9fc4ef-79c2-4016-881d-599a91f818e1\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") "
Sep 12 02:11:44.540270 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.540019    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") pod \"ca9fc4ef-79c2-4016-881d-599a91f818e1\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") "
Sep 12 02:11:44.540270 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.540071    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs\") pod \"ca9fc4ef-79c2-4016-881d-599a91f818e1\" (UID: \"ca9fc4ef-79c2-4016-881d-599a91f818e1\") "
Sep 12 02:11:44.540438 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.540418    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rwhd\" (UniqueName: \"kubernetes.io/projected/eec9c7d8-8e68-42f5-953a-07488d2b7504-kube-api-access-6rwhd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:44.540508 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.540439    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eec9c7d8-8e68-42f5-953a-07488d2b7504-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:44.540508 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.540418    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ca9fc4ef-79c2-4016-881d-599a91f818e1" (UID: "ca9fc4ef-79c2-4016-881d-599a91f818e1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:11:44.542103 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.542078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94" (OuterVolumeSpecName: "kube-api-access-ncz94") pod "ca9fc4ef-79c2-4016-881d-599a91f818e1" (UID: "ca9fc4ef-79c2-4016-881d-599a91f818e1"). InnerVolumeSpecName "kube-api-access-ncz94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:11:44.542180 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.542138    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ca9fc4ef-79c2-4016-881d-599a91f818e1" (UID: "ca9fc4ef-79c2-4016-881d-599a91f818e1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:11:44.542180 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.542159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ca9fc4ef-79c2-4016-881d-599a91f818e1" (UID: "ca9fc4ef-79c2-4016-881d-599a91f818e1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:11:44.641248 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.641213    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncz94\" (UniqueName: \"kubernetes.io/projected/ca9fc4ef-79c2-4016-881d-599a91f818e1-kube-api-access-ncz94\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:44.641248 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.641240    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca9fc4ef-79c2-4016-881d-599a91f818e1-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:44.641248 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.641250    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:44.641469 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:44.641259    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca9fc4ef-79c2-4016-881d-599a91f818e1-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:11:45.158591 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.158554    2582 generic.go:356] "Generic (PLEG): container finished" podID="eec9c7d8-8e68-42f5-953a-07488d2b7504" containerID="1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf" exitCode=0
Sep 12 02:11:45.158591 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.158596    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"
Sep 12 02:11:45.159148 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.158641    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" event={"ID":"eec9c7d8-8e68-42f5-953a-07488d2b7504","Type":"ContainerDied","Data":"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"}
Sep 12 02:11:45.159148 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.158696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc" event={"ID":"eec9c7d8-8e68-42f5-953a-07488d2b7504","Type":"ContainerDied","Data":"80e8bb32b24abf0ce593ad04f84d1e8138f077089489cdf57257d2062b6f058d"}
Sep 12 02:11:45.159148 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.158725    2582 scope.go:117] "RemoveContainer" containerID="1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"
Sep 12 02:11:45.160700 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.160672    2582 generic.go:356] "Generic (PLEG): container finished" podID="ca9fc4ef-79c2-4016-881d-599a91f818e1" containerID="bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a" exitCode=0
Sep 12 02:11:45.160804 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.160727    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"
Sep 12 02:11:45.160804 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.160748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" event={"ID":"ca9fc4ef-79c2-4016-881d-599a91f818e1","Type":"ContainerDied","Data":"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"}
Sep 12 02:11:45.160804 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.160793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t" event={"ID":"ca9fc4ef-79c2-4016-881d-599a91f818e1","Type":"ContainerDied","Data":"6681326bc41f09010cdf156d4779cdfe35f336a41614ddfd13200ce9e06dfb8c"}
Sep 12 02:11:45.168062 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.168023    2582 scope.go:117] "RemoveContainer" containerID="1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"
Sep 12 02:11:45.168350 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:11:45.168327    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf\": container with ID starting with 1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf not found: ID does not exist" containerID="1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"
Sep 12 02:11:45.168460 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.168356    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf"} err="failed to get container status \"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf\": rpc error: code = NotFound desc = could not find container \"1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf\": container with ID starting with 1ee81ced91d1017d6786feb9e7e84337ea30005acceffa3ffcfeafc104970cbf not found: ID does not exist"
Sep 12 02:11:45.168460 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.168371    2582 scope.go:117] "RemoveContainer" containerID="bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"
Sep 12 02:11:45.176510 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.176489    2582 scope.go:117] "RemoveContainer" containerID="bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"
Sep 12 02:11:45.176942 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:11:45.176905    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a\": container with ID starting with bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a not found: ID does not exist" containerID="bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"
Sep 12 02:11:45.177089 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.176965    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a"} err="failed to get container status \"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a\": rpc error: code = NotFound desc = could not find container \"bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a\": container with ID starting with bfb94164ccc18e43f9416b7a383a75c2a6a325d9759524e6bde32fd33f5e7c8a not found: ID does not exist"
Sep 12 02:11:45.178510 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.178486    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 02:11:45.181791 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.181766    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-98f68e-6478db4f46-8kb9t"]
Sep 12 02:11:45.192317 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.192287    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 02:11:45.200974 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:45.200949    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-98f68e-6d95d54ff4-22bmc"]
Sep 12 02:11:46.554697 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:46.554665    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca9fc4ef-79c2-4016-881d-599a91f818e1" path="/var/lib/kubelet/pods/ca9fc4ef-79c2-4016-881d-599a91f818e1/volumes"
Sep 12 02:11:46.555085 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:11:46.555052    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec9c7d8-8e68-42f5-953a-07488d2b7504" path="/var/lib/kubelet/pods/eec9c7d8-8e68-42f5-953a-07488d2b7504/volumes"
Sep 12 02:15:00.135162 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135085    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"]
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135463    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca9fc4ef-79c2-4016-881d-599a91f818e1" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135475    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca9fc4ef-79c2-4016-881d-599a91f818e1" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135485    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" containerName="payload-processing"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135491    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" containerName="payload-processing"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135497    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135502    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135509    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b72af814-cdb2-4fd5-b6df-a2527d1819be" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135515    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b72af814-cdb2-4fd5-b6df-a2527d1819be" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135521    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113f3614-0d19-4194-94e0-6b35dd15a014" containerName="main"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135526    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="113f3614-0d19-4194-94e0-6b35dd15a014" containerName="main"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135534    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135539    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135546    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="152472a3-44b0-4843-ba42-2ee526496455" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135551    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="152472a3-44b0-4843-ba42-2ee526496455" containerName="maas-api"
Sep 12 02:15:00.135554 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135561    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135567    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135572    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09f1ed73-07a0-4c35-a468-50e0e07e9523" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135577    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="09f1ed73-07a0-4c35-a468-50e0e07e9523" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135585    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec9c7d8-8e68-42f5-953a-07488d2b7504" containerName="payload-pre-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135590    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec9c7d8-8e68-42f5-953a-07488d2b7504" containerName="payload-pre-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135596    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135602    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135611    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135616    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135623    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135627    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135678    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135684    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b72af814-cdb2-4fd5-b6df-a2527d1819be" containerName="maas-api"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135692    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="113f3614-0d19-4194-94e0-6b35dd15a014" containerName="main"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135698    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="09f1ed73-07a0-4c35-a468-50e0e07e9523" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135703    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c027080d-04c5-4a2a-bd7a-b7f1b34e19ed" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="633d9bc5-d002-488f-80c6-c9f974b0de35" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135715    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0770b0-8ad1-46fe-8592-ebc1dfecf4aa" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135721    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135726    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="736c0762-6a3a-45fd-941a-aa2c0cd6d007" containerName="payload-processing"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135731    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="152472a3-44b0-4843-ba42-2ee526496455" containerName="maas-api"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135737    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca9fc4ef-79c2-4016-881d-599a91f818e1" containerName="maas-api"
Sep 12 02:15:00.136066 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.135742    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec9c7d8-8e68-42f5-953a-07488d2b7504" containerName="payload-pre-processing"
Sep 12 02:15:00.138584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.138569    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"
Sep 12 02:15:00.140711 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.140693    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7x88s\""
Sep 12 02:15:00.148801 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.148779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"
Sep 12 02:15:00.150391 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.150367    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"]
Sep 12 02:15:00.417173 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:00.417147    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"]
Sep 12 02:15:00.419064 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:15:00.419022    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfe1d12_bcd6_4441_ad94_81b8113aea2e.slice/crio-71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1 WatchSource:0}: Error finding container 71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1: Status 404 returned error can't find the container with id 71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1
Sep 12 02:15:01.051485 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:01.051448    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45" event={"ID":"4bfe1d12-bcd6-4441-ad94-81b8113aea2e","Type":"ContainerStarted","Data":"ab9154433c1e8fd9ce2b550ced1973dee326468f97c083a6964700b52239bd6f"}
Sep 12 02:15:01.053651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:01.051505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45" event={"ID":"4bfe1d12-bcd6-4441-ad94-81b8113aea2e","Type":"ContainerStarted","Data":"71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1"}
Sep 12 02:15:01.069193 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:01.069146    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45" podStartSLOduration=1.069130363 podStartE2EDuration="1.069130363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:15:01.067391397 +0000 UTC m=+1794.994337285" watchObservedRunningTime="2026-09-12 02:15:01.069130363 +0000 UTC m=+1794.996076278"
Sep 12 02:15:02.056684 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:02.056650    2582 generic.go:356] "Generic (PLEG): container finished" podID="4bfe1d12-bcd6-4441-ad94-81b8113aea2e" containerID="ab9154433c1e8fd9ce2b550ced1973dee326468f97c083a6964700b52239bd6f" exitCode=0
Sep 12 02:15:02.057097 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:02.056738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45" event={"ID":"4bfe1d12-bcd6-4441-ad94-81b8113aea2e","Type":"ContainerDied","Data":"ab9154433c1e8fd9ce2b550ced1973dee326468f97c083a6964700b52239bd6f"}
Sep 12 02:15:03.187644 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:03.187619    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"
Sep 12 02:15:04.067282 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:04.067254    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"
Sep 12 02:15:04.067467 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:04.067289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45" event={"ID":"4bfe1d12-bcd6-4441-ad94-81b8113aea2e","Type":"ContainerDied","Data":"71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1"}
Sep 12 02:15:04.067467 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:04.067333    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71747c3d5d38fab84b251ebffbd934f5de972076040360cb1478950f0edb56d1"
Sep 12 02:15:06.673060 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:06.673016    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:15:06.673541 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:06.673271    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:15:32.171060 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.171003    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:15:32.171563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.171536    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4bfe1d12-bcd6-4441-ad94-81b8113aea2e" containerName="cleanup"
Sep 12 02:15:32.171563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.171551    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4bfe1d12-bcd6-4441-ad94-81b8113aea2e" containerName="cleanup"
Sep 12 02:15:32.171658 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.171647    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4bfe1d12-bcd6-4441-ad94-81b8113aea2e" containerName="cleanup"
Sep 12 02:15:32.171695 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.171658    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b44d6bf5-5b27-4ccf-864a-a8648727856e" containerName="istio-proxy"
Sep 12 02:15:32.175233 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.175209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.177702 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.177678    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-103dc5-openshift-default-dockercfg-xqrrs\""
Sep 12 02:15:32.186841 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.186820    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:15:32.245660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245825 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245825 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb64r\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245825 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245781    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245835    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.245999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.245947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.246236 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.246001    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347223 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347394 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347394 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb64r\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347394 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347394 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347394 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347829 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347779    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347836    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.347954 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.347888    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.348145 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.348122    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.348252 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.348218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.349670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.349642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.349973 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.349954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.355278 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.355258    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.355412 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.355392    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb64r\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r\") pod \"e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") " pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.486820 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.486752    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:32.764584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.764558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:15:32.766226 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:15:32.766194    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod334a16a0_bba0_4786_bb8f_cae86d1c6307.slice/crio-82438430dc9c6cbdc344facbd68dc8328d3f8bd1c61c6cddb2b91978b2b4858d WatchSource:0}: Error finding container 82438430dc9c6cbdc344facbd68dc8328d3f8bd1c61c6cddb2b91978b2b4858d: Status 404 returned error can't find the container with id 82438430dc9c6cbdc344facbd68dc8328d3f8bd1c61c6cddb2b91978b2b4858d
Sep 12 02:15:32.768345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.768308    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:32.768444 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.768375    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:32.768444 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:32.768404    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:33.201475 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:33.201435    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerStarted","Data":"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"}
Sep 12 02:15:33.201844 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:33.201495    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerStarted","Data":"82438430dc9c6cbdc344facbd68dc8328d3f8bd1c61c6cddb2b91978b2b4858d"}
Sep 12 02:15:33.221031 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:33.220985    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podStartSLOduration=1.220969099 podStartE2EDuration="1.220969099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:15:33.219129249 +0000 UTC m=+1827.146075140" watchObservedRunningTime="2026-09-12 02:15:33.220969099 +0000 UTC m=+1827.147914984"
Sep 12 02:15:33.487121 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:33.487020    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:33.491641 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:33.491620    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:34.206384 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:34.206347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:34.207283 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:34.207260    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:36.526952 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.526916    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:15:36.530863 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.530835    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.533281 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.533252    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-103dc5-serving-cert\""
Sep 12 02:15:36.533439 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.533339    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-565nn\""
Sep 12 02:15:36.541399 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.540550    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:15:36.686873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.686830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.686873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.686875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.687182 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.686905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njnnm\" (UniqueName: \"kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.687182 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.687083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.788565 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.788466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.788565 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.788510    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.788565 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.788538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njnnm\" (UniqueName: \"kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.788844 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.788644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.788844 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:15:36.788814    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-serving-cert: secret "maas-api-e2e-ipp-a-103dc5-serving-cert" not found
Sep 12 02:15:36.788974 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:15:36.788906    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls podName:e4f728db-cafc-4fec-8fd9-61c51f02ac1d nodeName:}" failed. No retries permitted until 2026-09-12 02:15:37.288881035 +0000 UTC m=+1831.215826905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls") pod "maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" (UID: "e4f728db-cafc-4fec-8fd9-61c51f02ac1d") : secret "maas-api-e2e-ipp-a-103dc5-serving-cert" not found
Sep 12 02:15:36.789176 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.789158    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.791019 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.791001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:36.799282 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:36.799261    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njnnm\" (UniqueName: \"kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:37.292718 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:37.292682    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:37.295134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:37.295115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:37.446245 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:37.446207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:37.722634 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:37.722539    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:15:37.724706 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:15:37.724673    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f728db_cafc_4fec_8fd9_61c51f02ac1d.slice/crio-ab578fa4fadbfffe3df90f20d53daa61a77b9a5d83dc3ebc3cd2764b691c58e5 WatchSource:0}: Error finding container ab578fa4fadbfffe3df90f20d53daa61a77b9a5d83dc3ebc3cd2764b691c58e5: Status 404 returned error can't find the container with id ab578fa4fadbfffe3df90f20d53daa61a77b9a5d83dc3ebc3cd2764b691c58e5
Sep 12 02:15:37.726267 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:37.726247    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:15:38.226782 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:38.226745    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" event={"ID":"e4f728db-cafc-4fec-8fd9-61c51f02ac1d","Type":"ContainerStarted","Data":"ab578fa4fadbfffe3df90f20d53daa61a77b9a5d83dc3ebc3cd2764b691c58e5"}
Sep 12 02:15:39.233709 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:39.233674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" event={"ID":"e4f728db-cafc-4fec-8fd9-61c51f02ac1d","Type":"ContainerStarted","Data":"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"}
Sep 12 02:15:39.234144 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:39.233799    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:39.253013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:39.252957    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" podStartSLOduration=2.736119218 podStartE2EDuration="3.252942239s" totalImagesPullingTime="516.823021ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:15:39.252310284 +0000 UTC m=+1833.179256170" watchObservedRunningTime="2026-09-12 02:15:39.252942239 +0000 UTC m=+1833.179888125"
Sep 12 02:15:45.243644 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:45.243608    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:15:50.248208 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:50.248163    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 02:15:50.248587 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:50.248215    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 02:15:51.291822 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.291787    2582 generic.go:356] "Generic (PLEG): container finished" podID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerID="35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c" exitCode=0
Sep 12 02:15:51.292320 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.291832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerDied","Data":"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"}
Sep 12 02:15:51.292320 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.292202    2582 scope.go:117] "RemoveContainer" containerID="35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"
Sep 12 02:15:51.293221 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.293189    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:51.293284 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.293259    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:51.293320 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:51.293294    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:15:52.300495 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:52.300455    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerStarted","Data":"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"}
Sep 12 02:15:52.487477 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:52.487433    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:52.487683 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:52.487555    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 02:15:52.487683 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:52.487612    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 02:15:53.487715 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:53.487675    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 02:15:53.488285 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:53.488253    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 02:15:54.487219 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:54.487178    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 02:15:54.487432 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:54.487251    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 02:15:55.488086 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:55.488015    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 02:15:55.488615 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:55.488110    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 02:15:56.487491 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:56.487446    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Sep 12 02:15:56.487695 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:56.487515    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Sep 12 02:15:57.490910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:57.490879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:57.491411 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:57.491284    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:15:57.491971 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:15:57.491951    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:16:01.205304 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.205271    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:16:01.216896 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.216864    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.219602 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.219399    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-01570e-openshift-default-dockercfg-phdm2\""
Sep 12 02:16:01.220778 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.220751    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:16:01.341220 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341185    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341401 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341256    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341401 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341401 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341401 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341401    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z565\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341437    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.341577 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.341515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442396 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442396 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442468    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z565\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.442616 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.443011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.443011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442872    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.443011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.442993    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.443181 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.443127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.443310 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.443292    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.444812 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.444787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.445191 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.445033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.450088 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.450063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z565\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.450088 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.450083    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token\") pod \"e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") " pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.530527 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.530451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:01.804011 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.803986    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:16:01.806205 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:16:01.806174    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddace6989_3cca_43f2_901d_84b5f3133083.slice/crio-07b76b67057c5e93c9768da649cb2b578e77bcc92eee17417c2d46ea3715238e WatchSource:0}: Error finding container 07b76b67057c5e93c9768da649cb2b578e77bcc92eee17417c2d46ea3715238e: Status 404 returned error can't find the container with id 07b76b67057c5e93c9768da649cb2b578e77bcc92eee17417c2d46ea3715238e
Sep 12 02:16:01.808619 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.808585    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:01.808729 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.808648    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:01.808729 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:01.808676    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:02.350023 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.349985    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerStarted","Data":"e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4"}
Sep 12 02:16:02.350433 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.350052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerStarted","Data":"07b76b67057c5e93c9768da649cb2b578e77bcc92eee17417c2d46ea3715238e"}
Sep 12 02:16:02.371238 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.371195    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podStartSLOduration=1.3711794529999999 podStartE2EDuration="1.371179453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:16:02.368972156 +0000 UTC m=+1856.295918042" watchObservedRunningTime="2026-09-12 02:16:02.371179453 +0000 UTC m=+1856.298125342"
Sep 12 02:16:02.530966 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.530929    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:02.532732 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.532702    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:02.532866 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:02.532755    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:03.534911 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:03.534880    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:04.359517 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:04.359480    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:04.360450 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:04.360427    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:20.401184 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:20.401149    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 12 02:16:20.401660 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:20.401205    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 12 02:16:21.451480 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.451415    2582 generic.go:356] "Generic (PLEG): container finished" podID="dace6989-3cca-43f2-901d-84b5f3133083" containerID="e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4" exitCode=0
Sep 12 02:16:21.451823 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.451487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerDied","Data":"e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4"}
Sep 12 02:16:21.451866 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.451859    2582 scope.go:117] "RemoveContainer" containerID="e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4"
Sep 12 02:16:21.452865 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.452838    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:21.452925 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.452912    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:21.452965 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:21.452952    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:16:22.459769 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:22.459734    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerStarted","Data":"ea82e58d97779e7af71873192dbafbd6f39597fb5bc4a8c768f294ead34faf87"}
Sep 12 02:16:22.530747 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:22.530704    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:22.530944 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:22.530909    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:22.531028 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:22.530966    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:23.531758 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:23.531722    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:23.532171 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:23.531775    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:24.531667 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:24.531630    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:24.531847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:24.531690    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:25.531666 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:25.531630    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:25.531845 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:25.531686    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:26.531344 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:26.531309    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 12 02:16:26.531734 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:26.531365    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 12 02:16:27.535276 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:27.535242    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:27.535638 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:27.535593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:27.536201 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:27.536184    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:16:34.736954 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:16:34.736879    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:17:10.954276 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:10.954243    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:17:11.688649 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.688605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:17:11.692696 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.692674    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.695209 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.695191    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-848a2c-openshift-default-dockercfg-6r2xt\""
Sep 12 02:17:11.700920 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.700897    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:17:11.786941 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.786889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.786941 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.786942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.786986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787093    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787164    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787210 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd6h\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787485 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.787485 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.787287    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888469 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888498    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888898 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888664    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888898 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd6h\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.888898 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.888852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.889143 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.889113    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.889241 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.889180    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.889241 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.889229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.889345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.889277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.889345 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.889283    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.891063 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.891028    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.891159 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.891080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.896868 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.896847    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:11.897133 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:11.897115    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd6h\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h\") pod \"e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") " pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:12.004195 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.004109    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:12.278268 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.278239    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:17:12.280306 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:17:12.280277    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa92b3e_697c_4397_a845_a48412dbe2bb.slice/crio-bff69795614f1ea2d779660e409b58dce9be8166db885798933a86eacb0ad42c WatchSource:0}: Error finding container bff69795614f1ea2d779660e409b58dce9be8166db885798933a86eacb0ad42c: Status 404 returned error can't find the container with id bff69795614f1ea2d779660e409b58dce9be8166db885798933a86eacb0ad42c
Sep 12 02:17:12.282921 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.282885    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:12.283007 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.282954    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:12.283007 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.282986    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:12.704106 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.704070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerStarted","Data":"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"}
Sep 12 02:17:12.704284 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.704128    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerStarted","Data":"bff69795614f1ea2d779660e409b58dce9be8166db885798933a86eacb0ad42c"}
Sep 12 02:17:12.723172 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:12.723124    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podStartSLOduration=1.723107398 podStartE2EDuration="1.723107398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:17:12.72255028 +0000 UTC m=+1926.649496194" watchObservedRunningTime="2026-09-12 02:17:12.723107398 +0000 UTC m=+1926.650053283"
Sep 12 02:17:13.005250 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:13.005171    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:13.010625 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:13.010603    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:13.708747 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:13.708718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:13.709610 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:13.709589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:35.832152 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.832120    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerID="e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd" exitCode=0
Sep 12 02:17:35.832556 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.832190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerDied","Data":"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"}
Sep 12 02:17:35.832624 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.832608    2582 scope.go:117] "RemoveContainer" containerID="e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"
Sep 12 02:17:35.833701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.833672    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:35.833752 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.833740    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:35.833786 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:35.833772    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 12 02:17:36.838723 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:36.838688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerStarted","Data":"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"}
Sep 12 02:17:37.004599 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:37.004557    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:37.004836 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:37.004813    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:37.004890 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:37.004865    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:38.004701 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:38.004664    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:38.005123 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:38.004719    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:39.005211 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:39.005170    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:39.005595 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:39.005247    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:40.005342 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:40.005297    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:40.005753 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:40.005383    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:41.004737 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:41.004697    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:41.004962 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:41.004756    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:42.005360 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:42.005318    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Sep 12 02:17:42.005749 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:42.005385    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Sep 12 02:17:43.008271 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:43.008238    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:43.008734 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:43.008625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:43.009257 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:43.009234    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:53.101536 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:53.101496    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:17:53.101992 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:53.101759    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" containerID="cri-o://ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6" gracePeriod=30
Sep 12 02:17:58.011111 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.011067    2582 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 12 02:17:58.011474 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.011139    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 12 02:17:58.349794 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.349762    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:58.433250 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433213    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433268    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433299    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433328    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433356    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433413 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433391    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhd6h\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433540    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs\") pod \"8fa92b3e-697c-4397-a845-a48412dbe2bb\" (UID: \"8fa92b3e-697c-4397-a845-a48412dbe2bb\") "
Sep 12 02:17:58.433670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433625    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:17:58.433885 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433798    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:17:58.433947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433879    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:17:58.433947 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433815    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:17:58.434104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433953    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.434104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433966    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.434104 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.433993    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:17:58.435610 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.435586    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:17:58.435720 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.435629    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h" (OuterVolumeSpecName: "kube-api-access-zhd6h") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "kube-api-access-zhd6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:17:58.435878 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.435852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:17:58.435878 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.435853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8fa92b3e-697c-4397-a845-a48412dbe2bb" (UID: "8fa92b3e-697c-4397-a845-a48412dbe2bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535157    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535181    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhd6h\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-kube-api-access-zhd6h\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535192    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535201    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535209    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535217    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fa92b3e-697c-4397-a845-a48412dbe2bb-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.535228 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.535224    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fa92b3e-697c-4397-a845-a48412dbe2bb-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:17:58.956052 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.956013    2582 generic.go:356] "Generic (PLEG): container finished" podID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerID="ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6" exitCode=0
Sep 12 02:17:58.956242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.956098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerDied","Data":"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"}
Sep 12 02:17:58.956242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.956177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql" event={"ID":"8fa92b3e-697c-4397-a845-a48412dbe2bb","Type":"ContainerDied","Data":"bff69795614f1ea2d779660e409b58dce9be8166db885798933a86eacb0ad42c"}
Sep 12 02:17:58.956242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.956109    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"
Sep 12 02:17:58.956242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.956200    2582 scope.go:117] "RemoveContainer" containerID="ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"
Sep 12 02:17:58.965229 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.965166    2582 scope.go:117] "RemoveContainer" containerID="e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"
Sep 12 02:17:58.973170 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.973149    2582 scope.go:117] "RemoveContainer" containerID="ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"
Sep 12 02:17:58.973499 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:17:58.973476    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6\": container with ID starting with ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6 not found: ID does not exist" containerID="ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"
Sep 12 02:17:58.973610 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.973504    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6"} err="failed to get container status \"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6\": rpc error: code = NotFound desc = could not find container \"ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6\": container with ID starting with ac2b3070502d008c084e1f876a1bf08ba46623690e6ad27d544f73f8427d59d6 not found: ID does not exist"
Sep 12 02:17:58.973610 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.973521    2582 scope.go:117] "RemoveContainer" containerID="e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"
Sep 12 02:17:58.973816 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:17:58.973796    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd\": container with ID starting with e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd not found: ID does not exist" containerID="e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"
Sep 12 02:17:58.973878 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.973823    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd"} err="failed to get container status \"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd\": rpc error: code = NotFound desc = could not find container \"e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd\": container with ID starting with e3c81afa3906effc14e04d4f4bd452e7a8cb936b52d57941e97e6d6c6cc493bd not found: ID does not exist"
Sep 12 02:17:58.973940 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.973919    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:17:58.977124 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:17:58.977106    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-848a2c-openshift-default-dcdd4694d-z64ql"]
Sep 12 02:18:00.555142 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:00.555110    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" path="/var/lib/kubelet/pods/8fa92b3e-697c-4397-a845-a48412dbe2bb/volumes"
Sep 12 02:18:12.363276 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.363238    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:18:12.364205 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.364148    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" podUID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" containerName="maas-api" containerID="cri-o://c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60" gracePeriod=30
Sep 12 02:18:12.619858 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.619837    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:18:12.762444 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.762410    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca\") pod \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") "
Sep 12 02:18:12.762651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.762470    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njnnm\" (UniqueName: \"kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm\") pod \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") "
Sep 12 02:18:12.762651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.762504    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") pod \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") "
Sep 12 02:18:12.762651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.762522    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs\") pod \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\" (UID: \"e4f728db-cafc-4fec-8fd9-61c51f02ac1d\") "
Sep 12 02:18:12.762865 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.762827    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e4f728db-cafc-4fec-8fd9-61c51f02ac1d" (UID: "e4f728db-cafc-4fec-8fd9-61c51f02ac1d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:18:12.764724 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.764690    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e4f728db-cafc-4fec-8fd9-61c51f02ac1d" (UID: "e4f728db-cafc-4fec-8fd9-61c51f02ac1d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:18:12.764824 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.764755    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e4f728db-cafc-4fec-8fd9-61c51f02ac1d" (UID: "e4f728db-cafc-4fec-8fd9-61c51f02ac1d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:18:12.764866 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.764838    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm" (OuterVolumeSpecName: "kube-api-access-njnnm") pod "e4f728db-cafc-4fec-8fd9-61c51f02ac1d" (UID: "e4f728db-cafc-4fec-8fd9-61c51f02ac1d"). InnerVolumeSpecName "kube-api-access-njnnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:18:12.864173 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.864132    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:18:12.864173 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.864169    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njnnm\" (UniqueName: \"kubernetes.io/projected/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-kube-api-access-njnnm\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:18:12.864173 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.864179    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:18:12.864430 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:12.864188    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4f728db-cafc-4fec-8fd9-61c51f02ac1d-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:18:13.028286 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.028202    2582 generic.go:356] "Generic (PLEG): container finished" podID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" containerID="c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60" exitCode=0
Sep 12 02:18:13.028286 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.028255    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"
Sep 12 02:18:13.028466 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.028294    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" event={"ID":"e4f728db-cafc-4fec-8fd9-61c51f02ac1d","Type":"ContainerDied","Data":"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"}
Sep 12 02:18:13.028466 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.028351    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp" event={"ID":"e4f728db-cafc-4fec-8fd9-61c51f02ac1d","Type":"ContainerDied","Data":"ab578fa4fadbfffe3df90f20d53daa61a77b9a5d83dc3ebc3cd2764b691c58e5"}
Sep 12 02:18:13.028466 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.028380    2582 scope.go:117] "RemoveContainer" containerID="c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"
Sep 12 02:18:13.037612 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.037591    2582 scope.go:117] "RemoveContainer" containerID="c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"
Sep 12 02:18:13.037851 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:18:13.037832    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60\": container with ID starting with c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60 not found: ID does not exist" containerID="c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"
Sep 12 02:18:13.037917 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.037864    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60"} err="failed to get container status \"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60\": rpc error: code = NotFound desc = could not find container \"c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60\": container with ID starting with c1c9468cdb01a3282bc96466e8c385b63a7a48416e62d480effcab07e31bba60 not found: ID does not exist"
Sep 12 02:18:13.066440 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.066417    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:18:13.076424 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:13.076402    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-103dc5-6dbc9c9c8f-ps5zp"]
Sep 12 02:18:14.554987 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:18:14.554954    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" path="/var/lib/kubelet/pods/e4f728db-cafc-4fec-8fd9-61c51f02ac1d/volumes"
Sep 12 02:19:14.779909 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:14.779873    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:19:14.780408 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:14.780160    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy" containerID="cri-o://961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb" gracePeriod=30
Sep 12 02:19:20.025794 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.025767    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:19:20.137496 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137474    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137526    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137549    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137580    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137614    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137638    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137659    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xb64r\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137718    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.137910 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137760    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data\") pod \"334a16a0-bba0-4786-bb8f-cae86d1c6307\" (UID: \"334a16a0-bba0-4786-bb8f-cae86d1c6307\") "
Sep 12 02:19:20.138106 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.137932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:20.138106 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.138078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:19:20.138218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.138105    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.138218 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.138172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data" (OuterVolumeSpecName: "istio-data") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:20.138333 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.138264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:20.138333 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.138301    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:20.139894 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.139867    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r" (OuterVolumeSpecName: "kube-api-access-xb64r") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "kube-api-access-xb64r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:19:20.139999 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.139888    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:20.140168 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.140142    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token" (OuterVolumeSpecName: "istio-token") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:19:20.140296 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.140184    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "334a16a0-bba0-4786-bb8f-cae86d1c6307" (UID: "334a16a0-bba0-4786-bb8f-cae86d1c6307"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:19:20.239164 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239141    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239164 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239161    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239170    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239179    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb64r\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-kube-api-access-xb64r\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239188    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239196    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239204    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/334a16a0-bba0-4786-bb8f-cae86d1c6307-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.239300 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.239212    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/334a16a0-bba0-4786-bb8f-cae86d1c6307-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:20.353439 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.353412    2582 generic.go:356] "Generic (PLEG): container finished" podID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerID="961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb" exitCode=0
Sep 12 02:19:20.353559 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.353466    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"
Sep 12 02:19:20.353559 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.353498    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerDied","Data":"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"}
Sep 12 02:19:20.353645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.353586    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc" event={"ID":"334a16a0-bba0-4786-bb8f-cae86d1c6307","Type":"ContainerDied","Data":"82438430dc9c6cbdc344facbd68dc8328d3f8bd1c61c6cddb2b91978b2b4858d"}
Sep 12 02:19:20.353645 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.353615    2582 scope.go:117] "RemoveContainer" containerID="961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"
Sep 12 02:19:20.363133 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.363117    2582 scope.go:117] "RemoveContainer" containerID="35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"
Sep 12 02:19:20.370576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.370560    2582 scope.go:117] "RemoveContainer" containerID="961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"
Sep 12 02:19:20.370835 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:19:20.370813    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb\": container with ID starting with 961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb not found: ID does not exist" containerID="961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"
Sep 12 02:19:20.370917 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.370847    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb"} err="failed to get container status \"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb\": rpc error: code = NotFound desc = could not find container \"961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb\": container with ID starting with 961b0d14364917b6cf7462c62016d78206ba6dc0422d956d3d919463d6a07ebb not found: ID does not exist"
Sep 12 02:19:20.370917 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.370872    2582 scope.go:117] "RemoveContainer" containerID="35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"
Sep 12 02:19:20.371171 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:19:20.371155    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c\": container with ID starting with 35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c not found: ID does not exist" containerID="35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"
Sep 12 02:19:20.371237 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.371179    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c"} err="failed to get container status \"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c\": rpc error: code = NotFound desc = could not find container \"35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c\": container with ID starting with 35a10c103d75cae55e14d69af693da8a4ea40e2344853565dd84062031e35f6c not found: ID does not exist"
Sep 12 02:19:20.374936 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.374915    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:19:20.379614 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.379591    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-103dc5-openshift-default-57468bb85d-qrvzc"]
Sep 12 02:19:20.555336 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:20.555273    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" path="/var/lib/kubelet/pods/334a16a0-bba0-4786-bb8f-cae86d1c6307/volumes"
Sep 12 02:19:32.202722 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:32.202645    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:19:32.203127 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:32.202861    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy" containerID="cri-o://ea82e58d97779e7af71873192dbafbd6f39597fb5bc4a8c768f294ead34faf87" gracePeriod=30
Sep 12 02:19:37.434344 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.434315    2582 generic.go:356] "Generic (PLEG): container finished" podID="dace6989-3cca-43f2-901d-84b5f3133083" containerID="ea82e58d97779e7af71873192dbafbd6f39597fb5bc4a8c768f294ead34faf87" exitCode=0
Sep 12 02:19:37.434664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.434371    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerDied","Data":"ea82e58d97779e7af71873192dbafbd6f39597fb5bc4a8c768f294ead34faf87"}
Sep 12 02:19:37.434664 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.434447    2582 scope.go:117] "RemoveContainer" containerID="e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4"
Sep 12 02:19:37.449831 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:19:37.449807    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4\": container with ID starting with e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4 not found: ID does not exist" containerID="e15afe06f95629b8700d67d6e4c47c03ab688250ab7b8870bba20249cab395d4"
Sep 12 02:19:37.453232 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.453215    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:19:37.498209 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498141    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498209 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498201    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498249    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498271    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498286    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498308    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498375 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498364    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z565\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498392    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket\") pod \"dace6989-3cca-43f2-901d-84b5f3133083\" (UID: \"dace6989-3cca-43f2-901d-84b5f3133083\") "
Sep 12 02:19:37.498651 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498417    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data" (OuterVolumeSpecName: "istio-data") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:37.498754 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498708    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.498754 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498737    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:37.498922 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:19:37.498922 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.498911    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:37.499053 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.499017    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:37.500752 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.500720    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token" (OuterVolumeSpecName: "istio-token") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:19:37.500928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.500903    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565" (OuterVolumeSpecName: "kube-api-access-6z565") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "kube-api-access-6z565". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:19:37.500928 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.500915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:19:37.501013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.500950    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dace6989-3cca-43f2-901d-84b5f3133083" (UID: "dace6989-3cca-43f2-901d-84b5f3133083"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:19:37.599561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599512    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z565\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-kube-api-access-6z565\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599548    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599559    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599561 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599572    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dace6989-3cca-43f2-901d-84b5f3133083-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599584    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dace6989-3cca-43f2-901d-84b5f3133083-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599597    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dace6989-3cca-43f2-901d-84b5f3133083-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599610    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:37.599888 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:37.599621    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dace6989-3cca-43f2-901d-84b5f3133083-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:38.440799 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.440769    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"
Sep 12 02:19:38.441230 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.440808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5" event={"ID":"dace6989-3cca-43f2-901d-84b5f3133083","Type":"ContainerDied","Data":"07b76b67057c5e93c9768da649cb2b578e77bcc92eee17417c2d46ea3715238e"}
Sep 12 02:19:38.441230 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.440864    2582 scope.go:117] "RemoveContainer" containerID="ea82e58d97779e7af71873192dbafbd6f39597fb5bc4a8c768f294ead34faf87"
Sep 12 02:19:38.462611 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.462584    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:19:38.465991 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.465969    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-01570e-openshift-default-65489969bd-58sl5"]
Sep 12 02:19:38.555296 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:38.555269    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dace6989-3cca-43f2-901d-84b5f3133083" path="/var/lib/kubelet/pods/dace6989-3cca-43f2-901d-84b5f3133083/volumes"
Sep 12 02:19:43.015181 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.015149    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"]
Sep 12 02:19:43.017822 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.017795    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-a744d5-29819655-mtx45"]
Sep 12 02:19:43.044996 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.044544    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:19:43.044996 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.044825    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerName="payload-pre-processing" containerID="cri-o://217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677" gracePeriod=30
Sep 12 02:19:43.147981 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.147942    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.55:9004: connect: connection refused"
Sep 12 02:19:43.298393 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.298367    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:19:43.353963 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.353936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume\") pod \"bf81eea7-29b0-48c0-a027-abdffc079c18\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") "
Sep 12 02:19:43.354120 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.353978    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbww7\" (UniqueName: \"kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7\") pod \"bf81eea7-29b0-48c0-a027-abdffc079c18\" (UID: \"bf81eea7-29b0-48c0-a027-abdffc079c18\") "
Sep 12 02:19:43.354361 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.354334    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf81eea7-29b0-48c0-a027-abdffc079c18" (UID: "bf81eea7-29b0-48c0-a027-abdffc079c18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:19:43.355978 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.355953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7" (OuterVolumeSpecName: "kube-api-access-vbww7") pod "bf81eea7-29b0-48c0-a027-abdffc079c18" (UID: "bf81eea7-29b0-48c0-a027-abdffc079c18"). InnerVolumeSpecName "kube-api-access-vbww7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:19:43.455524 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.455487    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf81eea7-29b0-48c0-a027-abdffc079c18-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:43.455524 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.455515    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbww7\" (UniqueName: \"kubernetes.io/projected/bf81eea7-29b0-48c0-a027-abdffc079c18-kube-api-access-vbww7\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:19:43.466533 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.466507    2582 generic.go:356] "Generic (PLEG): container finished" podID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerID="217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677" exitCode=0
Sep 12 02:19:43.466658 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.466551    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"
Sep 12 02:19:43.466658 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.466602    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" event={"ID":"bf81eea7-29b0-48c0-a027-abdffc079c18","Type":"ContainerDied","Data":"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"}
Sep 12 02:19:43.466730 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.466679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf" event={"ID":"bf81eea7-29b0-48c0-a027-abdffc079c18","Type":"ContainerDied","Data":"cd78c612fc8c002a7284f467bd8f2ddc69c2f02b9d7a5b28daa8c785fb782f05"}
Sep 12 02:19:43.466730 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.466702    2582 scope.go:117] "RemoveContainer" containerID="217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"
Sep 12 02:19:43.476361 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.476343    2582 scope.go:117] "RemoveContainer" containerID="217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"
Sep 12 02:19:43.476612 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:19:43.476591    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677\": container with ID starting with 217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677 not found: ID does not exist" containerID="217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"
Sep 12 02:19:43.476677 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.476620    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677"} err="failed to get container status \"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677\": rpc error: code = NotFound desc = could not find container \"217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677\": container with ID starting with 217f5c0ed7cb467c60c281ac40b2f7a516b812fe108ab973f5d7d26358365677 not found: ID does not exist"
Sep 12 02:19:43.488908 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.488879    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:19:43.493343 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:43.493321    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-a744d5-84d64f8f6b-nrpsf"]
Sep 12 02:19:44.555330 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:44.555297    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4bfe1d12-bcd6-4441-ad94-81b8113aea2e" path="/var/lib/kubelet/pods/4bfe1d12-bcd6-4441-ad94-81b8113aea2e/volumes"
Sep 12 02:19:44.555723 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:19:44.555621    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" path="/var/lib/kubelet/pods/bf81eea7-29b0-48c0-a027-abdffc079c18/volumes"
Sep 12 02:20:00.742289 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:00.742178    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:20:00.743102 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:00.742441    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" podUID="45f5dff4-ba23-4665-b056-254fc7ead5df" containerName="payload-pre-processing" containerID="cri-o://a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67" gracePeriod=30
Sep 12 02:20:00.995706 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:00.995631    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:20:01.108205 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.108170    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrwts\" (UniqueName: \"kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts\") pod \"45f5dff4-ba23-4665-b056-254fc7ead5df\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") "
Sep 12 02:20:01.108379 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.108227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume\") pod \"45f5dff4-ba23-4665-b056-254fc7ead5df\" (UID: \"45f5dff4-ba23-4665-b056-254fc7ead5df\") "
Sep 12 02:20:01.108655 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.108624    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45f5dff4-ba23-4665-b056-254fc7ead5df" (UID: "45f5dff4-ba23-4665-b056-254fc7ead5df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:20:01.110273 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.110244    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts" (OuterVolumeSpecName: "kube-api-access-hrwts") pod "45f5dff4-ba23-4665-b056-254fc7ead5df" (UID: "45f5dff4-ba23-4665-b056-254fc7ead5df"). InnerVolumeSpecName "kube-api-access-hrwts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:20:01.209617 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.209584    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwts\" (UniqueName: \"kubernetes.io/projected/45f5dff4-ba23-4665-b056-254fc7ead5df-kube-api-access-hrwts\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:01.209617 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.209610    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45f5dff4-ba23-4665-b056-254fc7ead5df-plugins-config-volume\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:01.549841 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.549810    2582 generic.go:356] "Generic (PLEG): container finished" podID="45f5dff4-ba23-4665-b056-254fc7ead5df" containerID="a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67" exitCode=0
Sep 12 02:20:01.550013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.549849    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"
Sep 12 02:20:01.550013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.549897    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" event={"ID":"45f5dff4-ba23-4665-b056-254fc7ead5df","Type":"ContainerDied","Data":"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"}
Sep 12 02:20:01.550013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.549943    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz" event={"ID":"45f5dff4-ba23-4665-b056-254fc7ead5df","Type":"ContainerDied","Data":"4795d48dbf4237711eb41008a79d06ee88b6e638e898b9d5f144253a250735b9"}
Sep 12 02:20:01.550013 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.549966    2582 scope.go:117] "RemoveContainer" containerID="a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"
Sep 12 02:20:01.558965 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.558949    2582 scope.go:117] "RemoveContainer" containerID="a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"
Sep 12 02:20:01.559207 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:20:01.559190    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67\": container with ID starting with a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67 not found: ID does not exist" containerID="a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"
Sep 12 02:20:01.559256 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.559214    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67"} err="failed to get container status \"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67\": rpc error: code = NotFound desc = could not find container \"a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67\": container with ID starting with a638039ec67cb9c9cfd0e56ec803f7bc1c9c7f9347466ba821af0144326bdd67 not found: ID does not exist"
Sep 12 02:20:01.572559 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.572534    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:20:01.574204 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:01.574183    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-fbb6d9-85489bfc4-nw7sz"]
Sep 12 02:20:02.562480 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:02.562442    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45f5dff4-ba23-4665-b056-254fc7ead5df" path="/var/lib/kubelet/pods/45f5dff4-ba23-4665-b056-254fc7ead5df/volumes"
Sep 12 02:20:06.699296 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:06.699264    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:20:06.700778 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:06.700750    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:20:07.415454 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:07.415418    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:20:07.415757 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:07.415710    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy" containerID="cri-o://c474a7e03a656f75a671ced4a7ea92da30ee4ff7f67689dd7a3ce13da1b54b2b" gracePeriod=30
Sep 12 02:20:10.006076 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.006026    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 02:20:10.006447 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.006235    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" podUID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" containerName="manager" containerID="cri-o://e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf" gracePeriod=30
Sep 12 02:20:10.244348 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.244324    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 02:20:10.382348 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.382317    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert\") pod \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") "
Sep 12 02:20:10.382498 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.382383    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs\") pod \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") "
Sep 12 02:20:10.382498 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.382449    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4sxd\" (UniqueName: \"kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd\") pod \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\" (UID: \"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2\") "
Sep 12 02:20:10.384311 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.384282    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" (UID: "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:20:10.384424 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.384396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd" (OuterVolumeSpecName: "kube-api-access-l4sxd") pod "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" (UID: "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2"). InnerVolumeSpecName "kube-api-access-l4sxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:20:10.384493 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.384468    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" (UID: "01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:20:10.483736 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.483711    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-webhook-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:10.483736 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.483731    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:10.483873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.483741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4sxd\" (UniqueName: \"kubernetes.io/projected/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2-kube-api-access-l4sxd\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:10.598751 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.598722    2582 generic.go:356] "Generic (PLEG): container finished" podID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" containerID="e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf" exitCode=0
Sep 12 02:20:10.598885 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.598777    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq"
Sep 12 02:20:10.598885 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.598806    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" event={"ID":"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2","Type":"ContainerDied","Data":"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"}
Sep 12 02:20:10.598885 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.598867    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-kpwgq" event={"ID":"01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2","Type":"ContainerDied","Data":"94b6fef22a1361aff6b1d054bfcd0b51ff38e5447e2c50f17b73ca2598667beb"}
Sep 12 02:20:10.599068 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.598897    2582 scope.go:117] "RemoveContainer" containerID="e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"
Sep 12 02:20:10.607805 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.607784    2582 scope.go:117] "RemoveContainer" containerID="e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"
Sep 12 02:20:10.608088 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:20:10.608068    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf\": container with ID starting with e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf not found: ID does not exist" containerID="e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"
Sep 12 02:20:10.608154 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.608099    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf"} err="failed to get container status \"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf\": rpc error: code = NotFound desc = could not find container \"e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf\": container with ID starting with e74a5001c997b162bcd9514a68b5ab2a547d9e79f7f0265fc2931eec678133bf not found: ID does not exist"
Sep 12 02:20:10.639584 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.639525    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 02:20:10.644456 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:10.644435    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-kpwgq"]
Sep 12 02:20:12.555949 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.555914    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" path="/var/lib/kubelet/pods/01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2/volumes"
Sep 12 02:20:12.614291 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.614262    2582 generic.go:356] "Generic (PLEG): container finished" podID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerID="c474a7e03a656f75a671ced4a7ea92da30ee4ff7f67689dd7a3ce13da1b54b2b" exitCode=0
Sep 12 02:20:12.614611 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.614327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerDied","Data":"c474a7e03a656f75a671ced4a7ea92da30ee4ff7f67689dd7a3ce13da1b54b2b"}
Sep 12 02:20:12.614727 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.614670    2582 scope.go:117] "RemoveContainer" containerID="54e860885570ee9dd8e10da9a54c2ea22608bcb9b494c13dc83bef92424140ef"
Sep 12 02:20:12.674806 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.674776    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:20:12.804001 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.803927    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804217 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804189    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data" (OuterVolumeSpecName: "istio-data") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:20:12.804332 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804219    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804332 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804264    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804332 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804294    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804522 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804384    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804522 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804447    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5d84\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.804522 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804502    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804919    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.804955    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket\") pod \"b76dfbc5-3a08-4bfb-81dd-91115122220c\" (UID: \"b76dfbc5-3a08-4bfb-81dd-91115122220c\") "
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.805254    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-data\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.805274    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b76dfbc5-3a08-4bfb-81dd-91115122220c-istiod-ca-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.805435    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:20:12.805647 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.805593    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:20:12.806119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.805657    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:20:12.808002 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.807948    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token" (OuterVolumeSpecName: "istio-token") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:20:12.808289 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.808248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 12 02:20:12.808385 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.808300    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 12 02:20:12.808578 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.808552    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84" (OuterVolumeSpecName: "kube-api-access-l5d84") pod "b76dfbc5-3a08-4bfb-81dd-91115122220c" (UID: "b76dfbc5-3a08-4bfb-81dd-91115122220c"). InnerVolumeSpecName "kube-api-access-l5d84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:20:12.906251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906216    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-podinfo\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906240    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906249    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-credential-socket\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906251 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906259    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-envoy\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906267    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-istio-token\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906275    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b76dfbc5-3a08-4bfb-81dd-91115122220c-workload-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:12.906569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:12.906282    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5d84\" (UniqueName: \"kubernetes.io/projected/b76dfbc5-3a08-4bfb-81dd-91115122220c-kube-api-access-l5d84\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:20:13.620802 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:13.620775    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"
Sep 12 02:20:13.621249 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:13.620803    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd" event={"ID":"b76dfbc5-3a08-4bfb-81dd-91115122220c","Type":"ContainerDied","Data":"b085b24f69b7fac64756dd50ddbed06f823ad348b28beb22512a9e97aef4a29c"}
Sep 12 02:20:13.621249 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:13.620858    2582 scope.go:117] "RemoveContainer" containerID="c474a7e03a656f75a671ced4a7ea92da30ee4ff7f67689dd7a3ce13da1b54b2b"
Sep 12 02:20:13.641347 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:13.641324    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:20:13.646563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:13.646537    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-fbb6d9-openshift-default-7456875548-m8lgd"]
Sep 12 02:20:14.554668 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:20:14.554630    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" path="/var/lib/kubelet/pods/b76dfbc5-3a08-4bfb-81dd-91115122220c/volumes"
Sep 12 02:21:06.854083 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:06.854002    2582 scope.go:117] "RemoveContainer" containerID="ab9154433c1e8fd9ce2b550ced1973dee326468f97c083a6964700b52239bd6f"
Sep 12 02:21:11.520429 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520391    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520713    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520725    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520736    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" containerName="manager"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520742    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" containerName="manager"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520749    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" containerName="maas-api"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520753    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" containerName="maas-api"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520764    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f5dff4-ba23-4665-b056-254fc7ead5df" containerName="payload-pre-processing"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520770    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f5dff4-ba23-4665-b056-254fc7ead5df" containerName="payload-pre-processing"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520775    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520781    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520786    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520791    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520799    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520804    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520809    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520814    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520820    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520825    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520831    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerName="payload-pre-processing"
Sep 12 02:21:11.520828 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520837    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerName="payload-pre-processing"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520844    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520849    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520856    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520861    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520909    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4f728db-cafc-4fec-8fd9-61c51f02ac1d" containerName="maas-api"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520917    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520924    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520929    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520936    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="01126ec2-baf9-4e0f-8f60-3cb1a6b8f7e2" containerName="manager"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520940    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa92b3e-697c-4397-a845-a48412dbe2bb" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520945    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf81eea7-29b0-48c0-a027-abdffc079c18" containerName="payload-pre-processing"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520950    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f5dff4-ba23-4665-b056-254fc7ead5df" containerName="payload-pre-processing"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520955    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.521529 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.520960    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dace6989-3cca-43f2-901d-84b5f3133083" containerName="istio-proxy"
Sep 12 02:21:11.525205 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.525188    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.528792 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.528767    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 02:21:11.529119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.528824    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 02:21:11.529119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.528865    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 02:21:11.529119 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.528935    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d79dj\""
Sep 12 02:21:11.531655 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.531629    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:11.659270 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.659243    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xpdg\" (UniqueName: \"kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.659423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.659286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.659423 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.659349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.760456 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.760433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.760569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.760488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xpdg\" (UniqueName: \"kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.760569 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.760526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.762982 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.762950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.763105 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.762950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.770479 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.770426    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xpdg\" (UniqueName: \"kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg\") pod \"maas-controller-6f875f6cb9-s4kfk\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") " pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:11.837671 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:11.837649    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:12.106741 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.106720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:12.108424 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:21:12.108398    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76dddea6_113b_4f1c_b450_a10c71d690a1.slice/crio-e4df40a6394087e6acdd9b165c6a85fad5322adb8b30e8c54dba03c55a1e7653 WatchSource:0}: Error finding container e4df40a6394087e6acdd9b165c6a85fad5322adb8b30e8c54dba03c55a1e7653: Status 404 returned error can't find the container with id e4df40a6394087e6acdd9b165c6a85fad5322adb8b30e8c54dba03c55a1e7653
Sep 12 02:21:12.109765 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.109748    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:21:12.876679 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.876648    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" event={"ID":"76dddea6-113b-4f1c-b450-a10c71d690a1","Type":"ContainerStarted","Data":"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"}
Sep 12 02:21:12.877082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.876699    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" event={"ID":"76dddea6-113b-4f1c-b450-a10c71d690a1","Type":"ContainerStarted","Data":"e4df40a6394087e6acdd9b165c6a85fad5322adb8b30e8c54dba03c55a1e7653"}
Sep 12 02:21:12.877082 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.876755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:12.894553 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:12.894501    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" podStartSLOduration=1.450522041 podStartE2EDuration="1.894487937s" totalImagesPullingTime="443.965896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:21:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:21:12.892917062 +0000 UTC m=+2166.819862982" watchObservedRunningTime="2026-09-12 02:21:12.894487937 +0000 UTC m=+2166.821433822"
Sep 12 02:21:23.884174 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:23.884139    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:33.264266 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.264229    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:33.264670 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.264423    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" podUID="76dddea6-113b-4f1c-b450-a10c71d690a1" containerName="manager" containerID="cri-o://7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c" gracePeriod=30
Sep 12 02:21:33.504873 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.504851    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:33.542563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.542492    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert\") pod \"76dddea6-113b-4f1c-b450-a10c71d690a1\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") "
Sep 12 02:21:33.542563 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.542528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs\") pod \"76dddea6-113b-4f1c-b450-a10c71d690a1\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") "
Sep 12 02:21:33.542747 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.542607    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xpdg\" (UniqueName: \"kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg\") pod \"76dddea6-113b-4f1c-b450-a10c71d690a1\" (UID: \"76dddea6-113b-4f1c-b450-a10c71d690a1\") "
Sep 12 02:21:33.544578 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.544546    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "76dddea6-113b-4f1c-b450-a10c71d690a1" (UID: "76dddea6-113b-4f1c-b450-a10c71d690a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:21:33.544675 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.544572    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "76dddea6-113b-4f1c-b450-a10c71d690a1" (UID: "76dddea6-113b-4f1c-b450-a10c71d690a1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:21:33.544675 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.544594    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg" (OuterVolumeSpecName: "kube-api-access-6xpdg") pod "76dddea6-113b-4f1c-b450-a10c71d690a1" (UID: "76dddea6-113b-4f1c-b450-a10c71d690a1"). InnerVolumeSpecName "kube-api-access-6xpdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:21:33.644098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.644066    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xpdg\" (UniqueName: \"kubernetes.io/projected/76dddea6-113b-4f1c-b450-a10c71d690a1-kube-api-access-6xpdg\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:21:33.644098 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.644096    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-webhook-cert\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:21:33.644285 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.644108    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/76dddea6-113b-4f1c-b450-a10c71d690a1-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:21:33.971453 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.971420    2582 generic.go:356] "Generic (PLEG): container finished" podID="76dddea6-113b-4f1c-b450-a10c71d690a1" containerID="7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c" exitCode=0
Sep 12 02:21:33.971628 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.971466    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk"
Sep 12 02:21:33.971628 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.971503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" event={"ID":"76dddea6-113b-4f1c-b450-a10c71d690a1","Type":"ContainerDied","Data":"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"}
Sep 12 02:21:33.971628 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.971556    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-s4kfk" event={"ID":"76dddea6-113b-4f1c-b450-a10c71d690a1","Type":"ContainerDied","Data":"e4df40a6394087e6acdd9b165c6a85fad5322adb8b30e8c54dba03c55a1e7653"}
Sep 12 02:21:33.971628 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.971579    2582 scope.go:117] "RemoveContainer" containerID="7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"
Sep 12 02:21:33.982372 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.982338    2582 scope.go:117] "RemoveContainer" containerID="7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"
Sep 12 02:21:33.983332 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:21:33.983299    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c\": container with ID starting with 7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c not found: ID does not exist" containerID="7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"
Sep 12 02:21:33.983473 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.983446    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c"} err="failed to get container status \"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c\": rpc error: code = NotFound desc = could not find container \"7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c\": container with ID starting with 7440cf8ca19e5bc1a597846ae555ad538ca44d60d7b8f41241cd4f8e67127c4c not found: ID does not exist"
Sep 12 02:21:33.995502 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.995479    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:33.998735 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:33.998713    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-s4kfk"]
Sep 12 02:21:34.554847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:34.554816    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76dddea6-113b-4f1c-b450-a10c71d690a1" path="/var/lib/kubelet/pods/76dddea6-113b-4f1c-b450-a10c71d690a1/volumes"
Sep 12 02:21:35.047395 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047364    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-dmz2b"]
Sep 12 02:21:35.047708 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047696    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76dddea6-113b-4f1c-b450-a10c71d690a1" containerName="manager"
Sep 12 02:21:35.047708 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047709    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76dddea6-113b-4f1c-b450-a10c71d690a1" containerName="manager"
Sep 12 02:21:35.047793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047769    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="334a16a0-bba0-4786-bb8f-cae86d1c6307" containerName="istio-proxy"
Sep 12 02:21:35.047793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047777    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76dfbc5-3a08-4bfb-81dd-91115122220c" containerName="istio-proxy"
Sep 12 02:21:35.047793 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.047786    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76dddea6-113b-4f1c-b450-a10c71d690a1" containerName="manager"
Sep 12 02:21:35.050579 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.050560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.053109 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.053085    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 12 02:21:35.054054 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054026    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-d79dj\""
Sep 12 02:21:35.054134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054092    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 12 02:21:35.054134 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054120    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 12 02:21:35.054772 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44t5\" (UniqueName: \"kubernetes.io/projected/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-kube-api-access-s44t5\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.054862 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-metrics-certs\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.054931 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.054880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-webhook-cert\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.060656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.060631    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-dmz2b"]
Sep 12 02:21:35.155510 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.155482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44t5\" (UniqueName: \"kubernetes.io/projected/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-kube-api-access-s44t5\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.155663 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.155540    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-metrics-certs\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.155663 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.155573    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-webhook-cert\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.158081 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.158031    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-metrics-certs\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.158335 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.158311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-webhook-cert\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.164290 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.164266    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44t5\" (UniqueName: \"kubernetes.io/projected/48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1-kube-api-access-s44t5\") pod \"maas-controller-6f875f6cb9-dmz2b\" (UID: \"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1\") " pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.361030 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.360950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:35.634954 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.634920    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f875f6cb9-dmz2b"]
Sep 12 02:21:35.982078 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:35.981977    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b" event={"ID":"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1","Type":"ContainerStarted","Data":"c9d920aa51dcc1a3569e17d421cb66393bb6b9ddaf69d874452b5461672b3e26"}
Sep 12 02:21:36.987848 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:36.987808    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b" event={"ID":"48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1","Type":"ContainerStarted","Data":"4fce1fcf69fb798706a542d3efffaeaf41b8c95bac04e81bda16b762ec683720"}
Sep 12 02:21:36.988259 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:36.987920    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:21:37.007149 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:37.007102    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b" podStartSLOduration=1.5413065019999999 podStartE2EDuration="2.007088185s" totalImagesPullingTime="465.781683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:21:37.005665429 +0000 UTC m=+2190.932611314" watchObservedRunningTime="2026-09-12 02:21:37.007088185 +0000 UTC m=+2190.934034082"
Sep 12 02:21:47.994550 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:21:47.994515    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f875f6cb9-dmz2b"
Sep 12 02:25:06.725978 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:25:06.725947    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:25:06.727282 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:25:06.727262    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2spkx_685ab9df-4254-4f9e-8208-30d659158759/ovn-acl-logging/0.log"
Sep 12 02:26:16.749814 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.749780    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"]
Sep 12 02:26:16.753101 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.753078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.755452 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.755433    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-565nn\""
Sep 12 02:26:16.764081 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.764057    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"]
Sep 12 02:26:16.855938 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.855901    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-metrics-certs\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.856114 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.855947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xspqr\" (UniqueName: \"kubernetes.io/projected/3c256f06-66da-4698-9e04-9d39e65a0989-kube-api-access-xspqr\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.856114 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.856002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-maas-api-tls\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.856114 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.856098    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c256f06-66da-4698-9e04-9d39e65a0989-openshift-service-ca\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.956635 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.956601    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-metrics-certs\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.956635 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.956640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xspqr\" (UniqueName: \"kubernetes.io/projected/3c256f06-66da-4698-9e04-9d39e65a0989-kube-api-access-xspqr\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.956832 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.956662    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-maas-api-tls\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.956832 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.956707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c256f06-66da-4698-9e04-9d39e65a0989-openshift-service-ca\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.957440 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.957418    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c256f06-66da-4698-9e04-9d39e65a0989-openshift-service-ca\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.959121 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.959100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-maas-api-tls\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.959242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.959224    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c256f06-66da-4698-9e04-9d39e65a0989-metrics-certs\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:16.963974 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:16.963950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xspqr\" (UniqueName: \"kubernetes.io/projected/3c256f06-66da-4698-9e04-9d39e65a0989-kube-api-access-xspqr\") pod \"maas-api-86c4dff545-gkc4j\" (UID: \"3c256f06-66da-4698-9e04-9d39e65a0989\") " pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:17.067232 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:17.067165    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:17.351536 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:17.351448    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"]
Sep 12 02:26:17.355780 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:26:17.355735    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c256f06_66da_4698_9e04_9d39e65a0989.slice/crio-a99db50690e1e700ce62e4d86cc98e75d2296d44178ff8c2edcd67ba88f441ee WatchSource:0}: Error finding container a99db50690e1e700ce62e4d86cc98e75d2296d44178ff8c2edcd67ba88f441ee: Status 404 returned error can't find the container with id a99db50690e1e700ce62e4d86cc98e75d2296d44178ff8c2edcd67ba88f441ee
Sep 12 02:26:17.357232 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:17.357211    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 12 02:26:18.215822 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:18.215730    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j" event={"ID":"3c256f06-66da-4698-9e04-9d39e65a0989","Type":"ContainerStarted","Data":"eda84205bc25799f7060e322c751048c154b1f2278eadbacaa6438299ba2e33f"}
Sep 12 02:26:18.215822 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:18.215799    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j" event={"ID":"3c256f06-66da-4698-9e04-9d39e65a0989","Type":"ContainerStarted","Data":"a99db50690e1e700ce62e4d86cc98e75d2296d44178ff8c2edcd67ba88f441ee"}
Sep 12 02:26:18.216267 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:18.215828    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:18.234656 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:18.234608    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j" podStartSLOduration=1.720341512 podStartE2EDuration="2.234593325s" totalImagesPullingTime="514.251813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-12 02:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:26:18.232262216 +0000 UTC m=+2472.159208117" watchObservedRunningTime="2026-09-12 02:26:18.234593325 +0000 UTC m=+2472.161539253"
Sep 12 02:26:24.225661 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:24.225630    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-86c4dff545-gkc4j"
Sep 12 02:26:31.895565 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:31.895532    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 02:26:31.895953 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:31.895719    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" podUID="17d21534-a743-48e1-b3e2-269ed57d827c" containerName="maas-api" containerID="cri-o://de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36" gracePeriod=30
Sep 12 02:26:33.145016 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.144989    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 02:26:33.199386 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.199358    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") pod \"17d21534-a743-48e1-b3e2-269ed57d827c\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") "
Sep 12 02:26:33.199521 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.199392    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qsqgt\" (UniqueName: \"kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt\") pod \"17d21534-a743-48e1-b3e2-269ed57d827c\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") "
Sep 12 02:26:33.199521 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.199430    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca\") pod \"17d21534-a743-48e1-b3e2-269ed57d827c\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") "
Sep 12 02:26:33.199521 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.199464    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls\") pod \"17d21534-a743-48e1-b3e2-269ed57d827c\" (UID: \"17d21534-a743-48e1-b3e2-269ed57d827c\") "
Sep 12 02:26:33.199804 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.199775    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17d21534-a743-48e1-b3e2-269ed57d827c" (UID: "17d21534-a743-48e1-b3e2-269ed57d827c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 12 02:26:33.201556 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.201531    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17d21534-a743-48e1-b3e2-269ed57d827c" (UID: "17d21534-a743-48e1-b3e2-269ed57d827c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:26:33.201637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.201541    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "17d21534-a743-48e1-b3e2-269ed57d827c" (UID: "17d21534-a743-48e1-b3e2-269ed57d827c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 12 02:26:33.201637 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.201575    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt" (OuterVolumeSpecName: "kube-api-access-qsqgt") pod "17d21534-a743-48e1-b3e2-269ed57d827c" (UID: "17d21534-a743-48e1-b3e2-269ed57d827c"). InnerVolumeSpecName "kube-api-access-qsqgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 12 02:26:33.279903 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.279835    2582 generic.go:356] "Generic (PLEG): container finished" podID="17d21534-a743-48e1-b3e2-269ed57d827c" containerID="de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36" exitCode=0
Sep 12 02:26:33.279903 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.279878    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"
Sep 12 02:26:33.280077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.279906    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" event={"ID":"17d21534-a743-48e1-b3e2-269ed57d827c","Type":"ContainerDied","Data":"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"}
Sep 12 02:26:33.280077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.279957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg" event={"ID":"17d21534-a743-48e1-b3e2-269ed57d827c","Type":"ContainerDied","Data":"a1c34ebbbc5222aea880869afe645e0dfe35e4810da48967641bfce62c29811d"}
Sep 12 02:26:33.280077 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.279981    2582 scope.go:117] "RemoveContainer" containerID="de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"
Sep 12 02:26:33.290018 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.289998    2582 scope.go:117] "RemoveContainer" containerID="de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"
Sep 12 02:26:33.290319 ip-10-0-128-81 kubenswrapper[2582]: E0912 02:26:33.290296    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36\": container with ID starting with de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36 not found: ID does not exist" containerID="de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"
Sep 12 02:26:33.290416 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.290328    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36"} err="failed to get container status \"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36\": rpc error: code = NotFound desc = could not find container \"de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36\": container with ID starting with de28c7c4956c803eb867a4b46d2b98d56329e92707d40989b8baac3fcc499c36 not found: ID does not exist"
Sep 12 02:26:33.300847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.300784    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d21534-a743-48e1-b3e2-269ed57d827c-openshift-service-ca\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:26:33.300847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.300832    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-maas-api-tls\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:26:33.300847 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.300847    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d21534-a743-48e1-b3e2-269ed57d827c-metrics-certs\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:26:33.301070 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.300860    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsqgt\" (UniqueName: \"kubernetes.io/projected/17d21534-a743-48e1-b3e2-269ed57d827c-kube-api-access-qsqgt\") on node \"ip-10-0-128-81.ec2.internal\" DevicePath \"\""
Sep 12 02:26:33.302876 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.302855    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 02:26:33.305940 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:33.305922    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-77b9747d97-ztlmg"]
Sep 12 02:26:34.554994 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:34.554960    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d21534-a743-48e1-b3e2-269ed57d827c" path="/var/lib/kubelet/pods/17d21534-a743-48e1-b3e2-269ed57d827c/volumes"
Sep 12 02:26:52.189927 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.189893    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cvq8m_5551845c-0b0d-4b98-8f57-7372cef926ec/manager/0.log"
Sep 12 02:26:52.442532 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.442446    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f875f6cb9-dmz2b_48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1/manager/0.log"
Sep 12 02:26:52.559212 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.559186    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q6k8q_c1ac51ac-faec-41be-b0b9-a09e6b0d1c11/server/0.log"
Sep 12 02:26:52.680663 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.680627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r57lk_37e490e7-2fe5-4c9c-a74d-7df7389d7ff7/manager/0.log"
Sep 12 02:26:52.792317 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.792252    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-6mdxq_4549777b-9184-4dd4-8f85-787ad4280243/copy-manifests/0.log"
Sep 12 02:26:52.799022 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:52.798973    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-6mdxq_4549777b-9184-4dd4-8f85-787ad4280243/manager/0.log"
Sep 12 02:26:53.536184 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:53.536155    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-86c4dff545-gkc4j_3c256f06-66da-4698-9e04-9d39e65a0989/maas-api/0.log"
Sep 12 02:26:53.886388 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:53.886362    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-bk7j7_fb0334e8-2748-426f-bf56-86867b14619a/postgres/0.log"
Sep 12 02:26:55.896085 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:55.896027    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-djtnb_413ca503-d97d-4d54-9883-17eef7612b9e/discovery/0.log"
Sep 12 02:26:56.012849 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:56.012819    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-4v2nr_38422c3e-d557-4476-a945-4d6b9bb5f942/kube-auth-proxy/0.log"
Sep 12 02:26:56.242101 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:56.242004    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/1.log"
Sep 12 02:26:56.367114 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:56.367081    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kwgmc_cc47836b-c450-47e4-a909-1bcdbec3b59e/payload-pre-processing/0.log"
Sep 12 02:26:56.489242 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:56.489211    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:26:56.961461 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:56.961430    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-tbpwr_c7fd7858-e180-4092-a21e-ee2e0e2d8e94/main/0.log"
Sep 12 02:26:57.075756 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:57.075727    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-d89mh_f5af3bbd-d620-4637-97a2-33e6acab602e/main/0.log"
Sep 12 02:26:57.409160 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:26:57.409125    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f7jw77_38956ee2-20d5-4eaa-afcf-6a2913ae22ff/main/0.log"
Sep 12 02:27:06.369961 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:06.369868    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dzpgh_d7fff888-e442-4677-a3c9-9b5d490bb332/global-pull-secret-syncer/0.log"
Sep 12 02:27:06.536187 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:06.536147    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x5zgg_1c3ac547-fd72-474e-91ff-cc7123edffd7/konnectivity-agent/0.log"
Sep 12 02:27:06.555877 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:06.555842    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-81.ec2.internal_8c31d8c8119d4fdd6c9aac0791f891f3/haproxy/0.log"
Sep 12 02:27:13.003240 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:13.003209    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmssx_f5ac8580-c05e-4412-8f9e-163eed7c1eac/node-exporter/0.log"
Sep 12 02:27:13.021159 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:13.021128    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmssx_f5ac8580-c05e-4412-8f9e-163eed7c1eac/kube-rbac-proxy/0.log"
Sep 12 02:27:13.040638 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:13.040616    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pmssx_f5ac8580-c05e-4412-8f9e-163eed7c1eac/init-textfile/0.log"
Sep 12 02:27:17.062328 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.062294    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"]
Sep 12 02:27:17.062682 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.062643    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d21534-a743-48e1-b3e2-269ed57d827c" containerName="maas-api"
Sep 12 02:27:17.062682 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.062655    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d21534-a743-48e1-b3e2-269ed57d827c" containerName="maas-api"
Sep 12 02:27:17.062754 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.062725    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d21534-a743-48e1-b3e2-269ed57d827c" containerName="maas-api"
Sep 12 02:27:17.065635 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.065615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.068214 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.068192    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-59ltq\"/\"default-dockercfg-pg6t6\""
Sep 12 02:27:17.068214 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.068201    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59ltq\"/\"kube-root-ca.crt\""
Sep 12 02:27:17.068924 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.068907    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-59ltq\"/\"openshift-service-ca.crt\""
Sep 12 02:27:17.076246 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.076222    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"]
Sep 12 02:27:17.175149 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.175115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-proc\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.175324 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.175230    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-lib-modules\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.175324 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.175254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-sys\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.175450 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.175319    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6r7\" (UniqueName: \"kubernetes.io/projected/1d0d4307-5e89-4660-9513-ecb00977062b-kube-api-access-sc6r7\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.175450 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.175371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-podres\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276033 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.275996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-lib-modules\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-sys\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6r7\" (UniqueName: \"kubernetes.io/projected/1d0d4307-5e89-4660-9513-ecb00977062b-kube-api-access-sc6r7\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276123    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-podres\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-proc\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276179    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-lib-modules\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276216 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276185    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-sys\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276443 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276231    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-proc\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.276443 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.276237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d0d4307-5e89-4660-9513-ecb00977062b-podres\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.283576 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.283547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6r7\" (UniqueName: \"kubernetes.io/projected/1d0d4307-5e89-4660-9513-ecb00977062b-kube-api-access-sc6r7\") pod \"perf-node-gather-daemonset-j6mwh\" (UID: \"1d0d4307-5e89-4660-9513-ecb00977062b\") " pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.376086 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.376063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:17.644290 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:17.644211    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"]
Sep 12 02:27:17.648250 ip-10-0-128-81 kubenswrapper[2582]: W0912 02:27:17.648217    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1d0d4307_5e89_4660_9513_ecb00977062b.slice/crio-c009e060f73d592876e12183981588e7b1db260372d23f128be4f16a0bab4742 WatchSource:0}: Error finding container c009e060f73d592876e12183981588e7b1db260372d23f128be4f16a0bab4742: Status 404 returned error can't find the container with id c009e060f73d592876e12183981588e7b1db260372d23f128be4f16a0bab4742
Sep 12 02:27:18.477747 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.477706    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh" event={"ID":"1d0d4307-5e89-4660-9513-ecb00977062b","Type":"ContainerStarted","Data":"7928ae4ded7ff2d8ebc7d39affb3a8d98e76597464ee267da2f1a75f34323e09"}
Sep 12 02:27:18.478132 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.477763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh" event={"ID":"1d0d4307-5e89-4660-9513-ecb00977062b","Type":"ContainerStarted","Data":"c009e060f73d592876e12183981588e7b1db260372d23f128be4f16a0bab4742"}
Sep 12 02:27:18.478132 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.477867    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:18.494299 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.494248    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh" podStartSLOduration=1.494234507 podStartE2EDuration="1.494234507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-12 02:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-12 02:27:18.493149115 +0000 UTC m=+2532.420095001" watchObservedRunningTime="2026-09-12 02:27:18.494234507 +0000 UTC m=+2532.421180393"
Sep 12 02:27:18.534004 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.533978    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8kws_ede064d1-2366-41ee-a796-62ecab84e0f8/dns/0.log"
Sep 12 02:27:18.551432 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.551413    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n8kws_ede064d1-2366-41ee-a796-62ecab84e0f8/kube-rbac-proxy/0.log"
Sep 12 02:27:18.613993 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:18.613970    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mvbd7_2120641c-d0f4-41db-b3bc-734724f523b7/dns-node-resolver/0.log"
Sep 12 02:27:19.236607 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:19.236581    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4ndgk_0c82b21e-bb74-4edf-9d48-e8bafbdc3037/node-ca/0.log"
Sep 12 02:27:20.946278 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:20.946242    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-djtnb_413ca503-d97d-4d54-9883-17eef7612b9e/discovery/0.log"
Sep 12 02:27:20.968571 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:20.968544    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-4v2nr_38422c3e-d557-4476-a945-4d6b9bb5f942/kube-auth-proxy/0.log"
Sep 12 02:27:21.023683 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:21.023658    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/0.log"
Sep 12 02:27:21.028509 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:21.028479    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/1.log"
Sep 12 02:27:21.063694 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:21.063668    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kwgmc_cc47836b-c450-47e4-a909-1bcdbec3b59e/payload-pre-processing/0.log"
Sep 12 02:27:21.099968 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:21.099946    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:27:22.031081 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:22.031021    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ql2vm_1d8a84f0-e955-4ca5-a08d-b2391e5ca88c/serve-healthcheck-canary/0.log"
Sep 12 02:27:22.620994 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:22.620964    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-km8s9_a1e45e25-6a1a-4019-a55c-b4a68713ced9/kube-rbac-proxy/0.log"
Sep 12 02:27:22.638669 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:22.638643    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-km8s9_a1e45e25-6a1a-4019-a55c-b4a68713ced9/exporter/0.log"
Sep 12 02:27:22.657070 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:22.657029    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-km8s9_a1e45e25-6a1a-4019-a55c-b4a68713ced9/extractor/0.log"
Sep 12 02:27:24.497969 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:24.497944    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-59ltq/perf-node-gather-daemonset-j6mwh"
Sep 12 02:27:24.693120 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:24.693092    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cvq8m_5551845c-0b0d-4b98-8f57-7372cef926ec/manager/0.log"
Sep 12 02:27:25.126982 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:25.126948    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f875f6cb9-dmz2b_48d5642b-3c4b-464a-8a80-9d6b5eb3e5f1/manager/0.log"
Sep 12 02:27:25.148665 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:25.148638    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-q6k8q_c1ac51ac-faec-41be-b0b9-a09e6b0d1c11/server/0.log"
Sep 12 02:27:25.292353 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:25.292298    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-r57lk_37e490e7-2fe5-4c9c-a74d-7df7389d7ff7/manager/0.log"
Sep 12 02:27:25.312862 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:25.312830    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-6mdxq_4549777b-9184-4dd4-8f85-787ad4280243/manager/0.log"
Sep 12 02:27:25.330144 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:25.330124    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-578c78994-6mdxq_4549777b-9184-4dd4-8f85-787ad4280243/copy-manifests/0.log"
Sep 12 02:27:26.978146 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:26.978061    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xjwpd_5b196256-5b66-492a-bf34-4cd859982789/openshift-lws-operator/0.log"
Sep 12 02:27:28.056021 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.055987    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-djtnb_413ca503-d97d-4d54-9883-17eef7612b9e/discovery/0.log"
Sep 12 02:27:28.067773 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.067748    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-4v2nr_38422c3e-d557-4476-a945-4d6b9bb5f942/kube-auth-proxy/0.log"
Sep 12 02:27:28.104417 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.104394    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/0.log"
Sep 12 02:27:28.109250 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.109180    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/1.log"
Sep 12 02:27:28.137832 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.137804    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kwgmc_cc47836b-c450-47e4-a909-1bcdbec3b59e/payload-pre-processing/0.log"
Sep 12 02:27:28.167121 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:28.167091    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:27:30.558915 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.558881    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-djtnb_413ca503-d97d-4d54-9883-17eef7612b9e/discovery/0.log"
Sep 12 02:27:30.589769 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.589743    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69c99bdc57-4v2nr_38422c3e-d557-4476-a945-4d6b9bb5f942/kube-auth-proxy/0.log"
Sep 12 02:27:30.632238 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.632208    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/0.log"
Sep 12 02:27:30.637531 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.637505    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-qk9nh_fea56fa2-4cc6-4ed1-b839-a8ccad4825e3/istio-proxy/1.log"
Sep 12 02:27:30.666080 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.666029    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-kwgmc_cc47836b-c450-47e4-a909-1bcdbec3b59e/payload-pre-processing/0.log"
Sep 12 02:27:30.699248 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:30.699219    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-nmhx2_00bf8136-c7c1-4807-a6f6-1ff68fccaff3/payload-processing/0.log"
Sep 12 02:27:33.293169 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.293139    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/kube-multus-additional-cni-plugins/0.log"
Sep 12 02:27:33.310768 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.310742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/egress-router-binary-copy/0.log"
Sep 12 02:27:33.328297 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.328276    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/cni-plugins/0.log"
Sep 12 02:27:33.345335 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.345317    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/bond-cni-plugin/0.log"
Sep 12 02:27:33.364733 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.364711    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/routeoverride-cni/0.log"
Sep 12 02:27:33.382718 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.382641    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/whereabouts-cni-bincopy/0.log"
Sep 12 02:27:33.400350 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.400332    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r2blc_e2d2ce75-d3cd-41cc-8c67-70c3deeb0196/whereabouts-cni/0.log"
Sep 12 02:27:33.846884 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.846795    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nwflt_440dbf68-d46f-4711-9c48-cbd0ede84229/kube-multus/0.log"
Sep 12 02:27:33.958995 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.958965    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk2nm_4db996df-7bb9-42a4-9c57-497e46831b9e/network-metrics-daemon/0.log"
Sep 12 02:27:33.973034 ip-10-0-128-81 kubenswrapper[2582]: I0912 02:27:33.973007    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xk2nm_4db996df-7bb9-42a4-9c57-497e46831b9e/kube-rbac-proxy/0.log"